Debugging Elasticsearch Slow Queries: From Shard Contention to Circuit Breakers

Elasticsearch distributes a query to every shard in the index. If you have 50 shards on 5 nodes, each node handles 30 shard-level queries (including replicas). The search thread pool has a fixed number of threads (default 13 per node). When 30 requests hit simultaneously, 13 run and 17 queue up. Each query waits for a thread, and the total latency becomes queue_time + execution_time. This is the contention spiral: as queue grows, response times become unpredictable and often trigger client timeouts.

The fix is to reduce the number of shards per node. A good rule of thumb is 20-40 shards per GB of heap. For a 30GB heap node, that means 600-1200 shards max across all indices. But for query performance, the active shards per query matters more. Aim for 1-2 shards per node per query. With 5 nodes, that's 5-10 shards total. Reindex with `_split` or `_shrink` to get there.

Circuit breakers prevent out-of-memory errors by aborting requests that exceed a memory threshold. But they can also cause slow queries by forcing retries or dropping requests before they complete. The two most common offenders are the fielddata circuit breaker and the request circuit breaker. Fielddata breaker trips when an aggregation loads too many unique values into memory. Request breaker trips when the serialized query size exceeds 60% of heap (default).

To diagnose, check `GET /_nodes/stats/breaker` and look for `tripped` counters. If you see trips, you need to either reduce the query size (e.g., use `composite` aggregation for high-cardinality buckets) or increase the breaker limit (not recommended). Better to optimize the query: avoid `terms` aggregation on fields with millions of unique values, and use `sampler` or `diversified_sampler` for approximate counts.

Hot threads output gives you a stack trace of threads that have been running for a long time. It's invaluable for identifying where time is spent: parsing, fetching, merging, or GC. But it's a snapshot, not a trend. Use it to see if threads are stuck in `search` phase (query execution) or `fetch` phase (retrieving documents). If many threads are in `search`, the bottleneck is shard-level processing. If in `fetch`, it's network or disk I/O.

I always run hot threads with `interval=5s` and `threads=10` to capture the hottest threads. Then I look for patterns: repeating method calls like `org.elasticsearch.search.aggregations.bucket.terms` suggest aggregation issues. Repeated `org.elasticsearch.index.fielddata` indicate fielddata loading. Use this to guide your next step—profile the query or check circuit breakers.

The default search thread pool has `max=13` and `queue_size=1000`. Increasing `max` can improve throughput but also increases heap pressure because each thread holds its own query context. A common mistake is to set `max` to the number of cores. That can cause OOM if queries are memory-heavy. Instead, increase slowly and monitor heap usage.

A better approach is to reduce the number of concurrent queries by adding a coordinating node layer. Coordinating nodes handle the scatter-gather phase and can queue requests independently, shielding data nodes from client timeouts. Set `number_of_coordinating_only_nodes` to 1-2 per data node. Then tune the data node search thread pool to match the shard count: if each node handles 10 shards per query, you need at least 10 threads, but no more than 20 to leave room for indexing.

Aggregations are the most common cause of slow queries in production. The `_profile` API reveals how time is split between phases: `query`, `fetch`, `aggregate`, and `reduce`. The aggregate phase is where buckets are built. If you see high time in `aggregate` with a large number of unique terms, you have a high-cardinality problem. For example, a `terms` aggregation on a `user_id` field with 5 million unique values will load all those into memory.

Solutions: use `composite` aggregation for paginated buckets, or `cardinality` for approximate counts. Alternatively, pre-compute bucket counts at index time using a `runtime_field` or a scripted metric. If you must have exact counts, limit the number of buckets with `size` and use `shard_size` to avoid excessive per-shard loading. I've seen queries go from 10 seconds to 200ms by reducing `size` from 10000 to 100 and using `composite`.