The pattern
When a popular cache entry expires, every concurrent request misses at the same moment and all of them regenerate the same expensive value. The cache that was supposed to protect the backend now synchronizes load against it. Stampedes are a coordination bug: the fix is almost never more cache, it is controlling who is allowed to recompute.
( 01 )Symptoms
How this failure announces itself.
- warningLatency spikes at suspiciously regular intervals that line up with a TTL.
- warningDatabase or upstream connection pools saturate briefly and recover on their own.
- warningRetries and timeouts amplify the spike instead of smoothing it.
- warningCache hit rate looks healthy on average but collapses to zero for specific hot keys.
( 02 )First moves
The first ten minutes — establish facts before touching code.
- 1Correlate the latency spikes with cache expiry times for the hottest keys; a stampede shows up as misses clustered in the same second.
- 2Count concurrent regenerations for one hot key during a spike - more than one is the smoking gun.
- 3Measure the cost of a single miss (query time, upstream call time) to size the blast radius.
- 4Check whether retries from timed-out callers re-enter the same miss path and multiply the load.
( 03 )Where to look
The code and config that usually owns this bug.
- searchTTL configuration for hot keys - identical TTLs set at the same time expire at the same time.
- searchLocking or single-flight guards around regeneration - is anything stopping N callers from recomputing concurrently?
- searchFallback behavior on cache miss - does the code go straight to the expensive source with no coalescing or shedding?
- searchPer-key metrics - which keys are hot, how often they expire, and how expensive one miss is.
( 04 )Common fixes
Fix the cause, then make the regression impossible.
- buildAdd request coalescing (single-flight) so only one caller regenerates a key while the rest wait for its result.
- buildStagger TTLs with jitter so hot keys never expire in the same instant.
- buildServe stale-while-revalidate: return the expired value while one background refresh runs.
- buildProtect the upstream with a concurrency cap so a stampede degrades politely instead of cascading.
( 05 )Prove the fix
A fix you can't demonstrate is a guess. Close the loop.
- verifiedLoad-test the hot key across its expiry and confirm exactly one regeneration happens per expiry window.
- verifiedConfirm p99 latency through the expiry moment stays flat instead of spiking.
- verifiedDisable the cache in a test environment and confirm the upstream degrades within its capacity limits.