Redis hot key: diagnose one logical key dominating capacity

The symptoms

• •p99 or p99.9 command latency rises on a single shard or single instance while other shards remain flat, indicating skewed load rather than uniform pressure.
• •One Redis instance CPU core runs near 100 percent while sibling cores or sibling shard processes stay idle, the classic single-thread command loop saturation pattern.
• •Client-side timeouts, retries, or circuit-breaker trips concentrate on a small set of key prefixes seen in application logs or APM traces.
• •Cluster slot mapping reports one or a few slots as hot in CLUSTER SLOTS or in the cluster manager dashboard, while other slots stay under-subscribed.
• •Memory evictions, TTL expiry, or replication backlog pressure concentrate on the node hosting the hot key, even though overall memory usage is within budget.

Likely causes

• •A counter, leaderboard, rate-limit bucket, or session token key that all requests read or write at the same logical path, so traffic converges on one O(1) lookup that still serializes through the command loop.
• •A large hash, list, or sorted set that is repeatedly scanned by a hot read path, so per-request CPU cost is high even when request count is moderate.
• •A cache-aside key with a missing or synchronously-refreshing backing store, so every miss produces a thundering herd against the same key prefix.
• •A monotonic counter or timestamp-based key that all writers append to, creating a hot append or hot INBY pattern on a single key.
• •Mis-sharded keys caused by a hash tag that pins too many keys to one slot, collapsing intended fan-out into one logical destination.
• •Read replica imbalance where one replica serves most reads of a popular key because client routing or replica selection favors it.

First ten minutes

1. 01Capture the symptom boundary: record which shard, instance, or pod shows CPU or latency divergence, and note whether the issue is read-side, write-side, or both, because the dominant command type determines which counter to read next.
2. 02Run INFO commandstats on the suspect node and rank commands by usec_per_call and calls_per_sec, because a single command dominating CPU is the most direct evidence of a hot key.
3. 03Run SLOWLOG GET on the suspect node and bucket slow entries by key name, because repeated slow entries against the same key are strong evidence of a hot key rather than network or memory pressure.
4. 04Sample MONITOR output for a short window against a non-production mirror or a canary node, then aggregate by key, because direct observation of key frequency is the most reliable signal.
5. 05Compare cluster slot distribution using CLUSTER COUNTKEYSINSLOT on a sampled set of slots, because a skewed key count per slot amplifies the effect of a single hot key.
6. 06Cross-reference suspected keys against application request logs or APM trace attributes in the same time window, to confirm that observed key frequency matches client behavior rather than an internal job.

Evidence to collect

• •Timestamped p50, p99, and p99.9 latency per shard or instance, separated by read and write, to establish the skew direction.
• •Per-key frequency counts from CLIENT LIST recent command samples, SLOWLOG entries, or a short MONITOR capture, ranked from highest to lowest.
• •Per-command CPU share from INFO commandstats, specifically usec_per_call and calls_per_sec, to identify which operation is saturating the loop.
• •Cluster slot occupancy from CLUSTER COUNTKEYSINSLOT on a representative slot sample, to detect hash-tag pinching or unbalanced slot key counts.
• •Application-side evidence tying the suspicious key prefix to a feature flag, deployment, or traffic source, so the fix is targeted at the code path rather than at Redis.

Where to look

• •The Redis command loop boundary, observable as the single main thread of the suspect instance, because Redis processes commands serially and any hot path shows up there first.
• •The cluster slot boundary, observable through CLUSTER SLOTS or the cluster manager console, because a hot key that all clients route to creates a hot slot regardless of node count.
• •The client library connection boundary, observable as connection pool saturation or per-host latency in APM, because hot keys also stress the client side and can mask the server-side signal.
• •The cache-aside origin boundary, observable as the backing store or origin service that issues the keys, because a hot key often originates in how the application namespaces its cache.
• •The replica selection boundary, observable in the read-routing layer or proxy config, because uneven read distribution can concentrate reads on one replica even when many replicas exist.

Diagnostic steps

1. 01Confirm single-thread saturation by comparing the suspect instance CPU to its peers; if only one node is hot while siblings are cool, the cause is per-instance load skew consistent with a hot key.
2. 02Rank keys by access frequency using SLOWLOG GET, a short MONITOR capture on a non-production instance, or the maxmemory-policy eviction log; the top one to three keys by frequency are candidates.
3. 03Cross-reference the top keys with the application's key naming convention to identify the feature and code path, so the fix targets the source rather than the symptom.
4. 04Measure per-command cost from INFO commandstats; if a single command type accounts for most CPU, the hot key is also a hot command and the fix should address both.
5. 05Check for hash tags using pattern matching on the key, and confirm with CLUSTER KEYSLOT, because a hash tag can pin multiple logical keys to one slot and amplify skew.
6. 06Distinguish hot key from hot key prefix by sampling with key masks; if a whole prefix is hot, the cause is upstream routing, not a single key, and the mitigation is different.
7. 07Rule out generic causes: confirm that AOF or RDB child processes are not the CPU consumer, that replication backlog is not stalled, and that memory pressure is not triggering eviction storms, because each of these can mimic hot-key symptoms.
8. 08Decide the mitigation class, read fan-out, write sharding, or backing-store offload, based on which command type dominates, because each class has a different safety profile.

Common mistakes

• •Concluding that Redis is CPU-bound overall and scaling out the cluster, when adding nodes does not help because the hot key is pinned to one slot by the hash function or by a hash tag.
• •Assuming high memory usage is the cause, when the dominant signal is single-thread CPU saturation with memory within budget, which points to hot key rather than memory pressure.
• •Disabling MONITOR as a general rule but missing that a short, scoped MONITOR capture on a non-production instance is the most direct evidence of per-key frequency.
• •Treating the hot key as a single string when it is actually a hot prefix or a hot hash tag, and applying a per-key fix that leaves the underlying routing skew intact.
• •Adding a longer TTL or larger cache to absorb the load, when the load is fundamentally per-request CPU and a larger cache will not change per-request cost.
• •Moving the hot key to a separate Redis cluster without addressing the application's read amplification, which simply relocates the saturation point.

Safe fixes

• •Apply read fan-out only after evidence shows reads dominate and the hot key is read-mostly, by introducing a local in-process cache or CDN with a short TTL, and confirm that the application's staleness tolerance model permits the lag.
• •Apply write sharding only after evidence shows writes dominate, by splitting the key into N sub-keys with a hash suffix, having the client write to all N and read from all N, and choosing N so that single-key load falls below the per-instance saturation threshold.
• •Apply a backing-store offload only after evidence shows the hot key is a derived value, by computing the value in the application tier or a batch process and verifying with a feature flag that latency does not regress.
• •Apply hash-tag removal only after CLUSTER KEYSLOT confirms a pin, by changing the key naming convention, deploying with a dual-read-and-rewrite pattern, and verifying that key distribution normalizes across slots.
• •Throttle or jitter the client path only as a temporary measure, by adding request coalescing or jitter at the application layer, and document the temporary nature so the structural fix is not skipped.
• •Each fix must be conditional on one of the diagnostic steps above, and each must include a proof step before declaring success.

Prove the fix

1. 01Per-shard p99 latency for the previously hot shard returns to within 10 percent of the median shard p99, sustained over at least one full traffic cycle, indicating the skew has been removed.
2. 02Per-key frequency from a follow-up MONITOR or SLOWLOG sample shows the previously hot key no longer ranks in the top decile of keys, indicating load has been distributed.
3. 03Single-instance CPU on the previously hot node falls back to within 15 percent of sibling node CPU, sustained across a normal traffic window, indicating the command loop is no longer saturated.
4. 04Cluster slot distribution from CLUSTER COUNTKEYSINSLOT on a sample shows reduced standard deviation across slots, indicating the hash distribution is no longer skewed.
5. 05Application-level error rate, timeout rate, and circuit-breaker trip count for the affected code path return to pre-incident baselines, confirming the user-visible symptom is resolved.

Prevention and next steps

• •Establish a key-naming convention that avoids hash tags across unrelated keys, and review key schemas during code review so that future features do not reintroduce slot pinching.
• •Add continuous monitoring of per-shard CPU and per-shard p99 latency with skew alerts, so a hot key is detected before it causes user-visible latency.
• •Track top keys by access frequency from a periodic offline SLOWLOG or MONITOR sample, and review the ranking with each release so new hot keys are spotted early.
• •Document the cache-aside pattern used in the application, including miss behavior and refresh strategy, so engineers understand the implications of a missing key before deploying a new feature.
• •Pre-define a key-sharding helper or abstraction in the application layer, so that when a hot key is detected the structural fix is a localized change rather than a cross-cutting rewrite.

Safe commands and checks

INFO commandstats
SLOWLOG GET <count>
CLUSTER COUNTKEYSINSLOT <slot>
CLUSTER KEYSLOT <key>
CLUSTER SLOTS
MONITOR