What this usually means
Yellow cluster status with unassigned shards usually indicates that the cluster cannot assign replica shards to their target nodes. This is almost always a resource or configuration problem: disk space exceeded on the target node, a node went offline and its shards haven't been reassigned, the number of replicas is higher than available nodes, or there's a persistent allocation decider (like `_tier_preference` or `_shard_size`) blocking allocation. Less commonly, shard data corruption or a stuck recovery queue can cause permanent unassignment. The allocation explain API (`GET _cluster/allocation/explain`) is the single most important diagnostic tool — it gives a precise reason.
The first ten minutes — establish facts before touching code.
- 1Run `GET _cluster/health?pretty` to confirm status and count unassigned shards.
- 2Run `GET _cluster/allocation/explain?pretty` to get a specific reason for one unassigned shard.
- 3Check disk usage on all data nodes: `GET _cat/allocation?v` or `df -h` on each node.
- 4Review cluster settings: `GET _cluster/settings?include_defaults&flat_settings` for `cluster.routing.allocation.disk.watermark.*`.
- 5Check node roles: `GET _cat/nodes?v&h=id,name,node.role,master` to ensure enough data nodes are available.
- 6Look for index-level settings: `GET index_name/_settings` for `index.routing.allocation.require.*` or `index.number_of_replicas`.
The specific files, logs, configs, and dashboards that usually own this bug.
- search`GET _cluster/allocation/explain` — primary diagnostic for reason and decision details.
- search`GET _cat/allocation?v` — per-node disk usage and shard counts.
- search`GET _cluster/settings?include_defaults&flat_settings` — watermarks, allocation filters.
- search`GET _nodes/stats/fs` — detailed disk info per node path.
- search`GET index_name/_recovery?pretty` — check stuck recovery processes.
- search`/var/log/elasticsearch/cluster-name.log` — cluster state changes, node join/leave events.
- search`GET _cat/shards?h=index,shard,prirep,state,node,unassigned.reason` — quick unassigned shard overview.
Practical causes, not theory. These are the things you will actually find.
- warningDisk usage exceeds the `cluster.routing.allocation.disk.watermark.flood_stage` threshold (default 95%) — cluster forces read-only and blocks shard allocation.
- warningNumber of replicas set higher than available data nodes (e.g., replicas=2 on a 2-node cluster).
- warningNode left cluster and shards remain unassigned because allocation deciders (like `_tier_preference`) prevent reassignment.
- warningIndex-level allocation filters (`index.routing.allocation.require.*`) that exclude all current nodes.
- warningCorrupt shard data after a node crash that prevents recovery.
- warningCluster has a `cluster.routing.allocation.enable: none` or `index.routing.allocation.enable: none` set (often left from maintenance).
- warningShard queue stuck — a shard recovery fails repeatedly and gets stuck in the retry loop.
Concrete fix directions. Pick the one that matches your root cause.
- buildFree disk space on nodes: delete unused indices, increase disk capacity, or `curl -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{"transient":{"cluster.routing.allocation.disk.watermark.flood_stage":"98%"}}'` as a temporary lifeline.
- buildReduce replica count: `PUT index_name/_settings { "index.number_of_replicas": 1 }` (if cluster has only 2 nodes).
- buildReroute shards: `POST _cluster/reroute?retry_failed=true` to retry stuck allocations.
- buildAdjust allocation deciders: remove `cluster.routing.allocation.include._tier_preference` or add missing tier.
- buildManually allocate a shard: `POST _cluster/reroute { "commands": [{ "allocate_replica": { "index": "...", "shard": 0, "node": "node_name" } }] }` (use `allocate_empty_primary` only if data is lost).
- buildClear read-only index block: `PUT index_name/_settings { "index.blocks.read_only_allow_delete": null }` if flood stage triggered.
- buildRestart the cluster one node at a time if recovery queue is stuck (rare, but works).
A fix you cannot prove is a guess. Close the loop.
- verifiedRun `GET _cluster/health` and check `status` becomes green and `unassigned_shards` becomes 0.
- verifiedRun `GET index_name/_recovery?human` to confirm shards are fully recovered (stage: DONE).
- verifiedCheck `GET _cat/shards` — all shards should have a node assigned and state STARTED.
- verifiedVerify search results are complete: compare doc count before and after fix.
- verifiedMonitor disk usage: `GET _cat/allocation?v` — all nodes should have balanced usage below both watermarks.
- verifiedSimulate a node failure test: stop a data node and see if replicas reassign within expected timeout.
Things that make this bug worse or harder to find.
- warningNever `allocate_empty_primary` unless you have fully lost the original data and are willing to accept data loss.
- warningDon't set watermarks above 99% — you risk node filling up and kernel OOM killing Elasticsearch.
- warningAvoid changing `cluster.routing.allocation.disk.include_relocations: true` without understanding relocation overhead.
- warningDon't forget to clear `cluster.routing.allocation.enable: none` after maintenance — I've seen clusters stuck yellow for weeks.
- warningDo not manually reroute a primary shard to a node that already has a replica of the same shard (duplicate).
- warningNever ignore flood_stage read-only blocks — they protect against disk-full crashes but need to be resolved before data can be written.
Replica Shard Stuck Unassigned After Disk Full
Timeline
- 14:22PagerDuty alert: Cluster health yellow. 47 unassigned shards.
- 14:23Check health: unassigned_shards=47, status=yellow. Kibana dashboards showing incomplete data.
- 14:25Run GET _cluster/allocation/explain — reason: 'disk exceeded watermark flood stage' on node data-3
- 14:27Check _cat/allocation: data-3 at 96% used. Flood stage at 95%.
- 14:30SSH to data-3, run df -h: /data mounted at 96%. Clean up old logstash indices (delete 3 indices, ~200GB).
- 14:35Run POST _cluster/reroute?retry_failed=true — no immediate change.
- 14:40Set transient watermark to 98% via API to allow allocation while cleanup continues.
- 14:45Check health again: unassigned_shards=12, still yellow. Run allocation explain again: now 'unable to allocate because index has read-only block'.
- 14:47Remove read-only block: PUT _all/_settings {"index.blocks.read_only_allow_delete": null}
- 14:50Health check: unassigned_shards=0, status green. Recovery complete in 5 minutes.
- 15:00Verify search results: doc count matches pre-incident baseline. Set watermark back to 95%.
At 14:22, our monitoring platform triggered a high-priority alert: Elasticsearch cluster health turned yellow with 47 unassigned shards. I was the primary SRE on call. The initial panic came from the fact that our logging pipeline was writing critical production logs, and some indices were missing replica shards — meaning if a node failed, we'd lose data. I quickly opened Kibana and saw that the logs-index-2025.03.15 had only one active shard instead of two per primary.
My first step was to run the allocation explain API. It showed the classic 'disk exceeded watermark flood stage' on node data-3. I SSH'd into that node and confirmed with df -h that the /data mount was at 96% utilization. The flood stage watermark was at the default 95%, so Elasticsearch had set all indices on that node to read-only, preventing any new shard allocation. I immediately deleted three old, unneeded indices that were taking up about 200GB. At the same time, I raised the watermark temporarily to 98% via a transient cluster setting to allow shard movement while the cleanup was still in progress.
After the cleanup, I ran the reroute API to retry failed allocations. The unassigned shard count dropped to 12, but then stalled. Allocation explain now showed a different error: 'unable to allocate because index has read-only block'. The flood stage had set a read-only block on all indices that had shards on data-3. I removed that block with a simple settings update. Within minutes, the cluster reassigned the remaining shards, and health turned green. The root cause was a disk space management gap — we didn't have an automated cleanup policy for old indices. I later implemented a Curator job to delete indices older than 30 days.
Root cause
Disk usage on node data-3 exceeded 95% flood stage, causing Elasticsearch to block writes and prevent shard allocation. After freeing space, the remaining read-only block blocked final allocation.
The fix
Delete old indices to free disk space. Temporarily raise disk watermark to 98% to allow allocation. Remove the read-only index block via settings API. Then run reroute with retry_failed.
The lesson
Always set up automated index lifecycle management (ILM) or Curator to prevent disk from reaching flood stage. The allocation explain API tells you exactly which decider is blocking — use it before guessing.
The allocation explain API (`GET _cluster/allocation/explain`) is the single most useful tool for yellow cluster debugging. It returns a JSON object with the current allocation decision for an unassigned shard, including the `deciders` array that lists each allocation decider and whether it was `YES`, `NO`, or `THROTTLE`.
The output includes fields like `current_state`, `unassigned_info` (reason and time), `allocation_delay` (if any), and `decisions` which is a list of decider names (e.g., `disk_threshold`, `replica_after_primary_active`, `shards_limit`, etc.) with their explanation. If the reason is `INDEX_CREATED` or `CLUSTER_RECOVERED`, the shard is just waiting for normal allocation. If it's `NODE_LEFT` or `REINITIALIZED`, it might need a reroute. The key is to look for the first `NO` decision — that's the root cause.
Pro tip: If you have many unassigned shards, run the explain API multiple times with different shard IDs to see if the same decider is blocking all of them. You can specify a shard explicitly: `GET _cluster/allocation/explain { "index": "logs-2025.03.15", "shard": 0, "primary": false }`.
Elasticsearch has three disk watermark levels: `low` (default 85%), `high` (default 90%), and `flood_stage` (default 95%). The low watermark triggers shard relocation away from the node, the high watermark prevents new shards from being allocated to it, and the flood stage forces all indices on that node into read-only mode. This is a safety mechanism to prevent data loss from disk-full crashes, but it can cause widespread unassigned shards.
Common mistakes: setting watermarks too close together (e.g., low=90%, high=95%, flood=96%) leaves almost no buffer for relocation. Another mistake is setting watermarks as absolute values (e.g., `low: "100GB"`) without understanding the filesystem's total size — I've seen nodes with 500GB total and a 100GB low watermark that triggers too early. Also note that watermarks are checked periodically (by default every 30 seconds), so changes don't take effect immediately.
For clusters with large disks (multiple TB), consider using absolute values rather than percentages. For example, on a 4TB node, 85% means 3.4TB — that's a lot of data to relocate if the node is at 90%. A better approach: set `low: "500GB"` (meaning free space, not used space), `high: "200GB"`, `flood: "50GB"` to ensure you always have a healthy buffer.
When a shard fails to allocate, Elasticsearch retries it with exponential backoff (up to a maximum of 5 retries by default). However, if the underlying issue (like insufficient disk space) persists, the shard will keep failing and eventually be marked as `unassigned` with a `reasons` field like `ALLOCATION_FAILED`. These shards are not automatically retried further — you must manually trigger a reroute with `retry_failed=true`.
In production, I've seen situations where a temporary network partition causes multiple shards to fail allocation, and after the partition heals, the shards remain unassigned because the retry limit was reached. The fix is a simple `POST _cluster/reroute?retry_failed=true`. But be careful: if the root cause hasn't been resolved (e.g., a node is still down), the reroute will fail again and consume cluster resources.
Another pattern: shards stuck in `INITIALIZING` state for hours. This usually means a recovery is stuck. Check `GET index_name/_recovery` for shards with `stage` not `DONE` and a `source` node that is offline. In that case, use the Cluster Allocation Explain API to see if the recovery is blocked. Sometimes you need to manually cancel the recovery by restarting the node holding the primary shard (or the replica node).
In Elasticsearch 7.x+, the `_tier_preference` setting is automatically added to indices created after upgrading or in clusters with data tiers. If your cluster doesn't have nodes with the appropriate tier roles (e.g., `data_hot`, `data_content`), shards may not allocate. For example, an index with `_tier_preference: data_hot` will only allocate to nodes with the `data_hot` role. If no such node exists, shards remain unassigned.
Similarly, index-level routing allocation rules like `index.routing.allocation.require.rack_id: rack1` can cause unassigned shards if no node has `rack_id: rack1`. This often happens when nodes are decommissioned but the index settings aren't updated. To find these, look at the index settings: `GET index_name/_settings` and examine `index.routing.allocation.*` fields.
A quick fix: temporarily remove the filter by setting it to null: `PUT index_name/_settings { "index.routing.allocation.require.rack_id": null }`. But the proper fix is to update your node attributes or tier roles to match the expected values.
Frequently asked questions
What does 'unassigned_shards' mean in cluster health?
Unassigned shards are shards that have not been allocated to any node. In a yellow cluster, the primary shard is assigned but one or more replicas are not. This means you have no redundancy for those shards. The cluster health status is yellow if all primaries are assigned but some replicas are unassigned. It becomes red if a primary is unassigned (data unavailable).
How do I find which nodes have disk issues causing unassigned shards?
Run `GET _cat/allocation?v` — it shows per-node disk usage percentage and shard counts. Nodes with high usage (above the low watermark) are likely causing allocation blocks. You can also check `GET _nodes/stats/fs` for detailed filesystem stats.
Can I safely ignore a yellow cluster if all searches work?
No. Yellow means replicas are missing, leaving your data vulnerable to loss if the node hosting the primary fails. Also, yellow can be a precursor to red if node failures happen. Always investigate and resolve unassigned shards promptly.
What is the difference between 'allocate_replica' and 'allocate_empty_primary' in reroute?
'allocate_replica' assigns an existing replica shard to a node (requires node to have the shard data). 'allocate_empty_primary' creates an empty primary shard — this data-loss operation should only be used if the original shard is lost and you can tolerate losing the data. Never use it unless you've verified the data is irretrievable.
Why did my cluster turn yellow after adding a new node?
Adding a node can cause temporary yellow status as shards rebalance. This is normal. But if it persists, check if the new node has the correct roles and if disk watermarks or allocation filters are blocking it. Also ensure the node is not excluded by any `cluster.routing.allocation.exclude` settings.