Fix Amazon OpenSearch Service Cluster Yellow and Red Status

A logging cluster went from green to red overnight with no deployment, no traffic spike, and no alarms configured to catch it until search queries started returning partial results the next morning. The cause: a single data node had crossed the disk watermark threshold from normal log growth, OpenSearch had automatically blocked writes to protect the node, and one index’s primary shard happened to live entirely on that node with no replica to fall back on. Nothing dramatic happened — disk usage just grew past a threshold nobody was watching, and the cluster did exactly what it’s supposed to do when that happens.

OpenSearch cluster health issues are almost always resource pressure (disk, JVM heap, or shard count) expressing itself as an unassigned shard. Here’s how to find which resource and which shard.

The Problem

Cluster status reflects shard allocation health, not application-level errors:

Symptom	What It Means
Status: yellow	Some replica shards are unassigned — data is intact but not fully redundant
Status: red	Some primary shards are unassigned — some data is currently unqueryable
Writes rejected with `ClusterBlockException`	A node crossed the disk watermark and the cluster is in a read-only block
Slow queries with high JVM memory pressure	Heap usage near the circuit breaker limit, causing GC pauses or query rejection
Shard count warnings	Too many shards per node relative to available heap, hurting both stability and performance

Every one of these is the cluster protecting itself or reporting a resource constraint — not a random malfunction.

Why Does This Happen?

Disk watermark thresholds exceeded: OpenSearch has low (85%), high (90%), and flood-stage (95%) disk watermarks by default. Crossing the high watermark stops shard allocation to that node; crossing flood-stage forces indices with shards on that node into a read-only block, which is what actually stops writes.
Insufficient replica count or unlucky shard placement: If an index has zero replicas, losing the single node holding a primary shard makes that shard’s data unavailable until the node is recovered or restored from a snapshot — there’s no replica to promote.
JVM heap pressure from oversized shards or excessive shard count: Each shard costs a baseline amount of heap overhead regardless of its size. Too many small shards per node, or a small number of very large shards, both push heap pressure into circuit-breaker territory, causing query rejections and slow garbage collection pauses.
Node loss due to instance retirement or AZ issue without enough standby capacity: Multi-AZ deployments handle a single node/AZ loss gracefully only if there’s enough remaining capacity and correctly configured zone awareness — a cluster sized right at capacity with no headroom struggles to reallocate shards after a node loss.
Index lifecycle not managing old indices: Time-series/log indices (daily indices, for example) that are never rolled over, force-merged, or deleted accumulate indefinitely, quietly consuming disk and shard count until the cluster hits a watermark it was never sized to hit.
Snapshot or backup failures going unnoticed: If automated snapshots have been silently failing (due to an IAM role or S3 permission issue), a red cluster status becomes a genuine data-loss risk rather than a recoverable inconvenience, because there’s no recent restore point.

The Fix

Step 1: Check Cluster Health and Identify Unassigned Shards

curl -s "https://search-orders-logs-abc123.us-east-1.es.amazonaws.com/_cluster/health?pretty"

curl -s "https://search-orders-logs-abc123.us-east-1.es.amazonaws.com/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason" | grep UNASSIGNED

The unassigned.reason column tells you directly why — ALLOCATION_FAILED, NODE_LEFT, or INDEX_CREATED are all distinct problems with distinct fixes.

Step 2: Check Node-Level Disk Usage

aws opensearch describe-domain \
  --domain-name orders-logs \
  --query "DomainStatus.ClusterConfig"

curl -s "https://search-orders-logs-abc123.us-east-1.es.amazonaws.com/_cat/allocation?v&h=node,disk.percent,disk.used,disk.avail"

If any node is above 90%, that’s the immediate cause of allocation being blocked.

Step 3: Free Up Disk or Scale Storage

Delete or archive old indices you no longer need queryable:

curl -X DELETE "https://search-orders-logs-abc123.us-east-1.es.amazonaws.com/logs-2026.03.*"

Or increase EBS volume size on the domain, which OpenSearch Service applies with no downtime via blue/green:

aws opensearch update-domain-config \
  --domain-name orders-logs \
  --ebs-options EBSEnabled=true,VolumeType=gp3,VolumeSize=200

Step 4: Clear the Read-Only Block Once Disk Pressure Is Resolved

After freeing space, the cluster may still be in a stale read-only-allow-delete block — clear it explicitly:

curl -X PUT "https://search-orders-logs-abc123.us-east-1.es.amazonaws.com/*/_settings" \
  -H 'Content-Type: application/json' \
  -d '{"index.blocks.read_only_allow_delete": null}'

Step 5: Check JVM Memory Pressure

aws cloudwatch get-metric-statistics \
  --namespace AWS/ES \
  --metric-name JVMMemoryPressure \
  --dimensions Name=DomainName,Value=orders-logs Name=ClientId,Value=111122223333 \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 \
  --statistics Maximum

Sustained values above 75% indicate the cluster needs either more/larger nodes, or fewer/larger shards to reduce per-shard heap overhead.

Step 6: Audit Shard Count and Size Per Node

curl -s "https://search-orders-logs-abc123.us-east-1.es.amazonaws.com/_cat/shards?v&h=index,shard,prirep,store,node" | sort -k4 -h

A commonly cited guideline is keeping shard sizes between 10–50 GB and shard count per node proportional to available heap (roughly 20 shards per GB of heap as an upper bound) — reindex with fewer, larger shards if you’re well outside that range.

Step 7: Verify Replica Configuration on Critical Indices

curl -s "https://search-orders-logs-abc123.us-east-1.es.amazonaws.com/orders-current/_settings?pretty" | grep number_of_replicas

Set at least one replica for any index where availability matters more than storage cost:

curl -X PUT "https://search-orders-logs-abc123.us-east-1.es.amazonaws.com/orders-current/_settings" \
  -H 'Content-Type: application/json' \
  -d '{"index.number_of_replicas": 1}'

Step 8: Confirm Snapshots Are Actually Succeeding

aws opensearch describe-domain \
  --domain-name orders-logs \
  --query "DomainStatus.SnapshotOptions"

curl -s "https://search-orders-logs-abc123.us-east-1.es.amazonaws.com/_snapshot/_status"

If automated snapshots have been failing, fix the underlying IAM/S3 permission issue immediately — a red cluster with no recent restore point is a materially worse situation than a red cluster you can roll back.

Is This Safe?

The _cluster/health, _cat, and describe calls are read-only. Deleting old indices is destructive and irreversible unless you have a snapshot — confirm retention requirements before running bulk deletes. Increasing EBS volume size via update-domain-config is safe and applied with no downtime through OpenSearch Service’s blue/green deployment process, though it can take time to complete depending on cluster size. Clearing the read-only block should only be done after confirming disk pressure is actually resolved — clearing it prematurely just lets the cluster hit the same block again on the next write.

Key Takeaway

OpenSearch cluster status is a direct, honest signal about shard allocation health — yellow means under-replicated, red means primary data is unavailable, and both usually trace back to disk watermarks, missing replicas, or JVM heap pressure from too many or too-large shards. Watch disk usage and JVM memory proactively with CloudWatch alarms rather than discovering the problem when the cluster turns the color that means “I already blocked your writes to protect myself.”

Have questions or ran into a different OpenSearch Service issue? Connect with me on LinkedIn or X.