A session store backed by ElastiCache for Redis started returning intermittent connection refused errors after what looked like a routine node failure. The replica promoted in under 30 seconds, right on schedule — but the application kept failing for another four minutes because the Redis client library had cached the old primary’s endpoint and wasn’t configured to refresh cluster topology on connection error. Redis did its job perfectly. The client just never asked it what changed.

Redis failover problems split cleanly into two categories: the cluster’s own failover behavior, and the client’s ability to notice and adapt to it. Here’s how to check both.

The Problem

Failures cluster around promotion timing and client-side topology awareness:

Symptom What It Means
Failover takes much longer than expected Replica lag delaying promotion, or Multi-AZ not enabled
MOVED or ASK errors in cluster mode Client isn’t cluster-aware and doesn’t follow slot redirection
Connections fail after failover completes Client-side connection pool not refreshing topology or DNS
CLUSTERDOWN errors A slot range has no available node — often during a botched resharding or node replacement
High latency spikes with no CPU pressure Large key eviction, SWAPDB/FLUSHALL blocking, or a single slow command blocking the single-threaded event loop

Redis’s own failover is fast and well-tested; the failure mode almost always lives in how the client discovers and reacts to the change.

Why Does This Happen?

  • Multi-AZ with automatic failover not enabled: Without it, a primary node failure requires manual intervention or waits for the node to be replaced without automatic replica promotion — a much longer recovery window than most teams expect.
  • Client library not cluster-mode aware: In cluster mode enabled deployments, keys are distributed across hash slots. A client that doesn’t implement the Redis Cluster protocol will fail outright or mishandle MOVED/ASK redirection responses, especially after a resharding operation moves slots between nodes.
  • Connection pooling that doesn’t detect topology changes: Long-lived connections or a pool that only refreshes on explicit reconnect (not on error) will keep routing traffic to a node that’s no longer the primary for a given slot range.
  • Replica lag delaying failover promotion: ElastiCache waits for a replica to be reasonably caught up before promoting it (to limit data loss), so a replica that’s lagged due to a large SCAN, big key deletion, or replication buffer overflow takes longer to become failover-ready.
  • A single expensive command blocking the event loop: Redis is single-threaded for command execution. A KEYS * command, a large SORT, or an unbounded LRANGE against a huge list can block every other client for the duration — indistinguishable from a hang if you’re not looking at slow log data.
  • Reserved memory and eviction policy mismatch: If maxmemory-policy is set to noeviction and memory fills up, writes start failing with OOM command not allowed instead of evicting older keys — often mistaken for a connectivity problem because the error surfaces as a failed command, not a clear “out of memory” message in application logs.

The Fix

Step 1: Confirm Multi-AZ and Automatic Failover Are Enabled

aws elasticache describe-replication-groups \
  --replication-group-id orders-session-cache \
  --query "ReplicationGroups[0].{MultiAZ:MultiAZ,AutomaticFailover:AutomaticFailover}"

If AutomaticFailover shows disabled, enable it:

aws elasticache modify-replication-group \
  --replication-group-id orders-session-cache \
  --automatic-failover-enabled \
  --apply-immediately

Step 2: Check Replica Lag Before Trusting Failover Speed

aws cloudwatch get-metric-statistics \
  --namespace AWS/ElastiCache \
  --metric-name ReplicationLag \
  --dimensions Name=CacheClusterId,Value=orders-session-cache-002 \
  --start-time $(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 \
  --statistics Maximum

Step 3: Verify the Client Library Supports Cluster Mode

For cluster mode enabled replication groups, confirm the client is using a cluster-aware library (redis-py-cluster/redis-py with cluster support, Lettuce, ioredis with cluster: true) rather than a plain single-node client pointed at the configuration endpoint. Check for MOVED errors in application logs — their presence confirms the client is at least receiving redirection responses but not following them automatically.

Step 4: Confirm the Application Uses the Cluster Configuration Endpoint, Not a Node Endpoint

aws elasticache describe-replication-groups \
  --replication-group-id orders-session-cache \
  --query "ReplicationGroups[0].ConfigurationEndpoint"

Applications should connect to this single configuration endpoint (which handles topology discovery) rather than hardcoding an individual node’s endpoint, which becomes stale the moment that node’s role changes.

Step 5: Check for Blocking Commands in the Slow Log

aws elasticache describe-cache-clusters \
  --cache-cluster-id orders-session-cache-001 \
  --show-cache-node-info

Connect directly and check the slow log for expensive commands:

redis-cli -h orders-session-cache.abc123.clustercfg.use1.cache.amazonaws.com -p 6379 SLOWLOG GET 10

Replace any KEYS * usage with SCAN with a small COUNT, and audit large collection commands (LRANGE, SMEMBERS, HGETALL) against keys that may have grown unexpectedly large.

Step 6: Check Memory Pressure and Eviction Policy

aws cloudwatch get-metric-statistics \
  --namespace AWS/ElastiCache \
  --metric-name DatabaseMemoryUsagePercentage \
  --dimensions Name=CacheClusterId,Value=orders-session-cache-001 \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 \
  --statistics Maximum

If memory usage is consistently high and maxmemory-policy is noeviction, switch to an eviction policy appropriate for a cache workload (commonly allkeys-lru):

aws elasticache modify-cache-parameter-group \
  --cache-parameter-group-name orders-cache-params \
  --parameter-name-values "ParameterName=maxmemory-policy,ParameterValue=allkeys-lru"

Step 7: Test Failover Deliberately

aws elasticache test-failover \
  --replication-group-id orders-session-cache \
  --node-group-id 0001

Measure actual application-observed downtime during this test, not just the console’s reported failover completion time — that gap is exactly where client-side topology staleness shows up.

Is This Safe?

The describe and CloudWatch calls are read-only. Enabling automatic failover and changing the eviction policy via modify-cache-parameter-group are low-risk and take effect on the next maintenance window unless applied immediately. test-failover performs a real failover with a brief real interruption — run it against a non-production replication group first, or during a maintenance window if run against production, since it’s meant to validate real behavior, not simulate it.

Key Takeaway

ElastiCache Redis failover itself is fast and reliable when Multi-AZ is enabled — the recovery time your application actually experiences is almost always determined by whether the client is cluster-aware, connects to the configuration endpoint instead of a node endpoint, and refreshes its topology on connection error. Test failover deliberately and measure the client’s actual recovery time, not the console’s reported promotion time — they are frequently very different numbers.

Have questions or ran into a different ElastiCache issue? Connect with me on LinkedIn or X.