Cache stampedes: identify the expiry boundary that synchronized traffic

The symptoms

• •Origin QPS spikes sharply while cache hit ratio stays near its recent baseline, ruling out a global miss-rate drift
• •Origin p99 latency rises in lockstep with the spike, then drops back, suggesting a sudden burst rather than a sustained backlog
• •Cache miss logs cluster around one specific key rather than spreading across the keyspace
• •Origin connection pool or worker queue saturates, while cache-tier connection counts look normal
• •CPU, lock-wait, or row-contention metrics rise only on queries served by the suspected hot key
• •The spike starts within a small window after a TTL boundary, a deploy, or a memory-pressure eviction event

Likely causes

• •Hot key with a short, fixed TTL expires while many concurrent callers are holding requests for it
• •Cache-aside wrapper has no request coalescing or single-flight, so every caller races to refetch on miss
• •TTL jitter is absent, so per-instance or per-shard timers expire the same key at the same wall-clock instant
• •A rolling restart resets in-process expiry timers fleet-wide, synchronizing the next miss wave
• •Cache eviction under memory pressure evicts a hot key, collapsing many readers onto the origin at once
• •Pre-warm step skipped for a hot key after deploy, so the first wave is fully cold and concurrent
• •Downstream protection (circuit breaker, rate limit) tripped off earlier and was never re-enabled, removing back-pressure that used to throttle the burst

First ten minutes

1. 01Open the cache dashboard and the origin dashboard side by side; align timestamps to within seconds so the miss event lines up with the origin spike
2. 02Identify the top-N keys by miss count in the surge window and record their configured TTL and any jitter policy
3. 03Note the wall-clock start of the origin spike and compare it to the most recent TTL boundary of the top-missed key
4. 04Capture origin connection-pool utilization, queue depth, and worker wait time at the surge timestamp before they roll out of retention
5. 05Freeze write access to dashboards and config; do not change TTL, eviction policy, or cache contents while evidence is being collected
6. 06Page the cache-owner team for a shared timeline of deploys, restarts, and config changes that bracket the surge

Evidence to collect

• •Time-series of cache hits and misses per key covering at least one full TTL interval before and after the surge
• •Origin QPS and latency p50, p95, p99 at second resolution across the surge window
• •Configured TTL, jitter range, and refresh policy for each top-missed key
• •Trace or log samples showing concurrent origin fetches for the same key during the miss window
• •Cache client configuration: connection count, per-call timeout, single-flight or coalescing flag, retry policy

Where to look

• •Cache tier dashboards: hit/miss ratio, eviction counter, key TTL distribution, expired-vs-evicted breakdown
• •Origin tier: connection pool saturation, query latency, lock or row contention, thread-pool queue depth
• •Application logs: cache-wrapper debug lines or span tags such as cache_miss clustered under one parent operation
• •Distributed tracing: spans for the suspected key showing overlapping fetch windows instead of serialized waits
• •Deploy, config-change, and feature-flag history bracketing the surge timestamp

Diagnostic steps

1. 01Plot miss count for the suspected key against wall time and look for a step change that aligns with a TTL boundary; this is the stampede fingerprint
2. 02Count distinct concurrent in-flight origin fetches for that key during the surge window using traces or structured logs
3. 03Compare origin QPS during the spike to QPS immediately before; the ratio approximates the number of callers that raced into the miss
4. 04Read the cache wrapper source or config and confirm whether request coalescing or single-flight is enabled and active on this code path
5. 05Check whether the key is read by services that share no local in-process cache, which would force them all to consult the shared store on miss
6. 06Verify whether TTL is set to a single value or a jittered range, and whether any layer rewrites the TTL on read
7. 07Rule out memory-pressure eviction by checking the evicted-keys counter and the allocator's recent pressure metrics
8. 08Cross-reference deploy and restart events with the surge start to see if a fleet-wide timer reset synchronized the expiry

Common mistakes

• •Scrolling origin capacity in response to the spike, which hides the synchronized miss instead of removing it
• •Raising TTL on the hot key without first proving the miss is TTL-aligned, masking the real cause behind a longer window
• •Issuing a bulk cache delete to "fix" freshness, which collapses every reader into the cold path and deepens the stampede
• •Adding aggressive retry on miss, multiplying origin load for the same key instead of dampening it
• •Blaming the cache tier when memory-pressure eviction is the actual trigger; the cache is obeying its eviction policy
• •Disabling a circuit breaker that was protecting the origin, removing back-pressure that previously kept the burst survivable

Safe fixes

• •Enable request coalescing or single-flight in the cache-aside wrapper so only one caller fetches the origin value per miss window
• •Add small TTL jitter on the hot key so expiries spread across a window rather than aligning to one instant
• •Introduce probabilistic early refresh so the entry is repopulated before it expires, keeping the miss path cold
• •Wrap the miss path in a short-lived lock with an explicit timeout, scoped to the cache layer and never held across network calls that bypass it
• •Pre-warm known hot keys after deploys or restarts so the first wave is not fully cold and concurrent

Prove the fix

1. 01Replay the surge scenario or wait for the next natural expiry, and observe that concurrent origin fetches for the key drop toward one per miss window
2. 02Confirm cache hit ratio returns to its prior baseline within one TTL interval after the change is rolled out
3. 03Verify origin p99 latency returns to its pre-incident band within the same observation window
4. 04Inspect new traces for the key: a single fetch span with peer requests waiting on its result, instead of overlapping fetches
5. 05Re-check the TTL distribution for the key and confirm jitter is actually applied rather than overridden by a later write

Prevention and next steps

• •Define a default TTL jitter policy in cache key templates and enforce it in code review for any new key
• •Require single-flight or coalescing in any cache-aside wrapper that reaches production, and unit-test the miss path for it
• •Track miss-concurrency per hot key as an SLO and alert when it exceeds a small threshold above one
• •Run load tests that simulate cold cache and synchronized expiry as part of the standard pre-deploy suite
• •Document a pre-warm checklist for hot keys and run it after any fleet-wide restart or config change to the cache layer

Safe commands and checks

redis-cli -h <cache-host> -p 6379 TTL <key>
redis-cli -h <cache-host> -p 6379 --scan --pattern '<key-pattern>' | head -n 20
redis-cli -h <cache-host> -p 6379 INFO stats | grep -E 'keyspace_hits|keyspace_misses|expired_keys|evicted_keys'
redis-cli -h <cache-host> -p 6379 CONFIG GET maxmemory-policy
ss -tan state established '( sport = :6379 or dport = :6379 )' | wc -l
pidstat -p <pid> -r -u 1 5
curl -s http://<metrics-endpoint>/metrics | grep -E 'cache_(hit|miss)|origin_request' | head -n 40
strace -p <pid> -e trace=network -c -f 2>&1 | head -n 40