Building a Redis Cache Layer for Session Failover

Our HCIS application runs on a JBoss EAP 7.4 cluster behind HAProxy. For most of its life, sessions were managed by the JBoss session replication mechanism - sticky sessions with a fallback. It worked well enough until it didn't. When a node failed mid-session, clinicians got logged out. In a hospital setting, that's not just annoying. It interrupts workflows that matter.

This is the story of how I built a Redis-backed distributed cache layer to fix that, what I got right, and the two things that bit me harder than expected.

The architecture goal

The requirement was straightforward on paper: if any node in the cluster fails, users should survive the failure without losing their session. They might see a brief interruption - an extra second while the request gets re-routed - but they should not be forced to log in again and lose their context.

The solution was to externalise session state. Instead of each JBoss node holding sessions in memory, every node reads and writes session data from a shared Redis instance. HAProxy handles routing. JBoss handles the application. Redis holds the state.

HAProxy (load balancer)
    │
    ├─── JBoss Node 1 ──┐
    └─── JBoss Node 2 ──┼──→ Redis (shared session store)
         JBoss Node N ──┘

When Node 1 fails, HAProxy re-routes the next request to Node 2. Node 2 reads the session from Redis and picks up exactly where Node 1 left off. The user notices nothing beyond a brief pause.

The implementation

JBoss EAP 7.4 supports Infinispan as its distributed cache provider. We configured Infinispan to use a Redis-compatible remote cache store via the Hotrod client. Session serialization happens automatically - JBoss serializes the HttpSession object and Infinispan writes it to Redis as a binary blob keyed by session ID.

TTL configuration was important. We set session expiry at the Redis level to match the application's session timeout, with a small buffer to account for clock skew between nodes:

redis.expiry = applicationTimeout + 30s

This ensures Redis doesn't evict sessions that are still active, but also guarantees eventual cleanup when sessions genuinely expire. Without the buffer, a race condition between the JBoss timeout check and the Redis TTL expiry can cause valid sessions to disappear.

What went wrong: the cache header conflict

The first issue we hit was subtle and took a few days to isolate. Some responses were sending conflicting cache headers - Cache-Control: no-store from the application security filter and Cache-Control: max-age=3600 from a legacy JSP page that had explicit cache headers baked in.

This didn't break session failover directly. But HAProxy, configured with sticky sessions as a fallback, was occasionally caching the session affinity decision incorrectly when it saw max-age headers. This caused some requests to get routed to a node that no longer had the session in its local hot cache, forcing a Redis lookup every time - slowing things down and creating confusing logs that made the problem look like a Redis connectivity issue.

The fix was to normalise cache headers in a servlet filter that runs before any response is written. Simple in retrospect. Expensive to diagnose.

What went wrong: serialization

The second issue was more fundamental. Not everything we were storing in HttpSession was serializable. Legacy code had been putting non-serializable objects - open JDBC connections, Hibernate sessions, instances of classes without implements Serializable - directly into the session object. With in-memory replication, this was silently ignored. With Redis-backed serialization, it threw NotSerializableException at runtime.

The audit took longer than the fix. We had to trace every session.setAttribute() call across the codebase, identify the non-serializable objects, and either fix their classes or redesign how that data was being held in session. In three cases, the right answer was to not store the object in session at all - it should have been fetched fresh on each request rather than cached.

If you're planning a similar migration, run serialization checks in a staging environment before touching production. The errors will all surface. Better there than at 2am.

The result

After both issues were resolved, the failover behaviour worked exactly as designed. We simulated node failures by hard-killing JBoss processes mid-session and confirmed that HAProxy re-routed cleanly and sessions survived. The Redis TTL cleanup worked correctly under load. Mean session read latency added roughly 8–12ms compared to local cache, which was acceptable.

The bigger benefit wasn't just failover. Externalising sessions also made deployments cleaner - we could now do rolling restarts without worrying about session loss, which had previously required scheduled downtime. That turned out to matter more day-to-day than the failure scenario we originally designed for.