Fix Aurora and RDS Multi-AZ Failover Not Triggering or Taking Too Long

We once watched an Aurora cluster’s writer instance fail a health check, sit unhealthy for four minutes, and never fail over — while the application kept happily sending writes to an endpoint that was quietly timing out. The cluster endpoint hadn’t repointed because the failover itself hadn’t been triggered; a misconfigured parameter group had disabled automatic failover on that specific instance months earlier during a “temporary” maintenance change that never got reverted. Multi-AZ is only as reliable as its last configuration review.

Failover problems are dangerous precisely because they’re invisible until the moment you need them. Here’s how to verify failover actually works, and diagnose it when it doesn’t.

The Problem

Symptoms show up as extended outages during what should be a transparent failover:

Symptom	What It Means
Failover doesn’t trigger on instance failure	Automatic failover disabled, or no eligible replica in a different AZ
Failover triggers but takes minutes, not seconds	DNS caching by the client, or replica lag delaying promotion
Application errors persist after failover completes	Connection pool holding stale TCP connections to the old writer IP
Manual failover succeeds, automatic doesn’t	Health check thresholds configured too leniently
Aurora Global Database doesn’t fail over across regions	Global failover requires a manual or Route 53-driven action — it is not automatic by default

The database layer failed over correctly in most of these cases — the application layer just didn’t notice.

Why Does This Happen?

Automatic failover disabled at the cluster or instance level: Multi-AZ deployments and Aurora clusters both have a setting that controls whether failover happens automatically. If it was toggled off for a maintenance window and never restored, the cluster silently loses its safety net.
No viable failover target: Aurora needs a reader replica in a different AZ with compatible capacity to promote. If all replicas were scaled down, deleted, or are in the same AZ as the writer, there’s nothing to fail over to.
Client-side DNS caching: The Aurora/RDS endpoint’s DNS record has a short TTL (typically 5 seconds) specifically so clients pick up the new writer IP quickly. JVM-based applications and some connection poolers cache DNS resolutions far longer than the TTL, ignoring it entirely, and keep connecting to the old IP.
Connection pool not configured to validate or recycle connections: Even after DNS updates, a pool holding long-lived idle connections to the old writer will keep handing out broken connections until they’re evicted or validated.
Replica lag delaying promotion: RDS/Aurora waits for the replica to catch up (or accepts some data loss window) before promoting. A replica that’s badly lagged due to a long-running query or replication bottleneck takes proportionally longer to become failover-ready.
Aurora Global Database cross-region failover is manual by design: Unlike single-region Multi-AZ, promoting a secondary region in an Aurora Global Database is a deliberate action (failover-global-cluster) to avoid an unplanned cross-region promotion during a transient network blip.

The Fix

Step 1: Confirm Automatic Failover Is Enabled

aws rds describe-db-clusters \
  --db-cluster-identifier orders-cluster \
  --query "DBClusters[0].{MultiAZ:MultiAZ,Status:Status}"

For each cluster member, check its failover priority — 0 is promoted first:

aws rds describe-db-clusters \
  --db-cluster-identifier orders-cluster \
  --query "DBClusters[0].DBClusterMembers[*].{Instance:DBInstanceIdentifier,Writer:IsClusterWriter,Priority:PromotionTier}" \
  --output table

For classic RDS Multi-AZ (non-Aurora):

aws rds describe-db-instances \
  --db-instance-identifier orders-db \
  --query "DBInstances[0].MultiAZ"

Step 2: Verify a Viable Failover Target Exists

Confirm at least one reader is in a different AZ than the writer and has a low promotion tier:

aws rds describe-db-clusters \
  --db-cluster-identifier orders-cluster \
  --query "DBClusters[0].DBClusterMembers[*].{Instance:DBInstanceIdentifier,AZ:DBClusterParameterGroupStatus}"

aws rds describe-db-instances \
  --query "DBInstances[?DBClusterIdentifier=='orders-cluster'].{Instance:DBInstanceIdentifier,AZ:AvailabilityZone}" \
  --output table

Step 3: Check Replica Lag Before Trusting Failover Speed

aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name AuroraReplicaLag \
  --dimensions Name=DBInstanceIdentifier,Value=orders-cluster-reader-1 \
  --start-time $(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 \
  --statistics Maximum

Lag consistently over a few hundred milliseconds means the replica will take longer to become failover-eligible — investigate the query causing it before you need a real failover.

Step 4: Test With a Controlled Manual Failover

Don’t wait for a real outage to discover failover doesn’t work — test it deliberately:

aws rds failover-db-cluster \
  --db-cluster-identifier orders-cluster \
  --target-db-instance-identifier orders-cluster-reader-1

Time how long the application actually loses connectivity, not just how long the console shows “failing over.”

Step 5: Fix Client-Side DNS Caching

For JVM applications, disable indefinite DNS caching:

# In application startup or java.security
networkaddress.cache.ttl=5

For connection poolers (HikariCP, PgBouncer, RDS Proxy clients), set a maximum connection lifetime shorter than any plausible failover window — 60 to 120 seconds — so stale connections are recycled proactively rather than relying on failure detection.

Step 6: Put RDS Proxy in Front of the Cluster

RDS Proxy handles connection draining and re-establishment transparently during failover, which removes most of the client-side caching problem entirely:

aws rds create-db-proxy \
  --db-proxy-name orders-proxy \
  --engine-family MYSQL \
  --auth '[{"AuthScheme":"SECRETS","SecretArn":"arn:aws:secretsmanager:us-east-1:111122223333:secret:orders-db-secret"}]' \
  --role-arn arn:aws:iam::111122223333:role/rds-proxy-role \
  --vpc-subnet-ids subnet-0abc123 subnet-0def456

Step 7: For Aurora Global Database, Confirm the Cross-Region Runbook

Cross-region failover is intentionally manual:

aws rds failover-global-cluster \
  --global-cluster-identifier orders-global \
  --target-db-cluster-identifier arn:aws:rds:us-west-2:111122223333:cluster:orders-cluster-secondary

Document and rehearse this command before an actual regional event — it is not something you want to be reading documentation for during an outage.

Is This Safe?

The describe and CloudWatch metric calls are read-only. Triggering failover-db-cluster as a test is a real failover with real (brief) write unavailability — do it in a maintenance window or against a non-production cluster first. failover-global-cluster is a significant action that repoints your primary region and should be treated as an operational event requiring the same coordination as a real disaster recovery exercise, not a routine check.

Key Takeaway

Multi-AZ and Aurora failover are only useful if you’ve verified they actually work — a disabled setting, a missing cross-AZ replica, or a badly lagged replica can all turn an automatic failover into a multi-minute outage. Test failover deliberately, put RDS Proxy or short connection lifetimes in front of your application to eliminate DNS caching surprises, and remember that Aurora Global Database’s cross-region failover is a manual action you need a rehearsed runbook for, not a switch that flips itself.

Have questions or ran into a different Aurora or RDS failover issue? Connect with me on LinkedIn or X.