Cache stampede lock bypass: identify concurrent refill paths

The symptoms

• •Two or more backend logs or traces show near-simultaneous origin fetches for the same cache key within a sub-second window, while downstream metrics show a single cache miss event for that key.
• •Origin load spikes during cold-cache or post-eviction intervals that exceed the expected single-refill cost, despite a lock primitive being configured in the cache layer.
• •Cache hit rate oscillates between periods of normal fill and short bursts of duplicated work, often coinciding with TTL expiration clusters or manual cache invalidations.
• •Lock-protected paths still produce duplicated SQL, RPC, or downstream API calls in traces, indicating the lock attempt never executed or never blocked the parallel caller.
• •Metric counters for lock acquisition, contention, or wait time show zero or missing values even though contention is observable in traces, suggesting the lock call path is not on the refill hot path.

Likely causes

• •Lock acquisition is performed outside the refill critical section, so cache lookup happens before the lock is checked and the lock only governs a no-op or a post-fetch write.
• •Multiple cache client instances are used by the same service (different serializers, key prefixes, or connection pools), and each writes to a separate effective key namespace, defeating single-key coordination.
• •Lock key construction uses a non-deterministic component (random nonce, per-process instance ID, truncated hash) so contenders do not collide on the same lock key and each believes it is the sole owner.
• •Lock TTL is shorter than the origin refill duration, so the owner releases or loses the lock before completing the write-back; followers then proceed without observing ownership state.
• •The refill path uses a code path that bypasses the wrapper function, such as an admin tool, a warmup job, a background prefetch, or a direct DAO call that reads the origin without consulting the cache facade.
• •Cache backend semantics differ from the lock backend (e.g., lock stored in Redis but cache stored in a separate cluster), and a race allows one caller to bypass the lock while still writing the cache.

First ten minutes

1. 01Confirm scope: identify the affected cache key, the application surface that issued duplicate refills, and the time window over which the stampede was observable in dashboards.
2. 02Pull two or three sampled traces from the stampede window and list, for each, the cache miss event, any lock acquisition record, the origin call, and the cache write event with their relative timestamps.
3. 03Compare the cached key value used for the lookup against the lock key value used for coordination; mismatched key construction is a primary bypass vector.
4. 04Inspect application configuration for the cache client, the lock client, and any key prefixing or namespace settings that could cause the same logical entity to map to different lock keys.
5. 05Search the codebase for any path that reads the origin data type without going through the documented cache-aside or stampede-protected wrapper, including background jobs and warmup scripts.
6. 06Record which component owns the lock primitive, the TTL configured on the lock, and whether the configured TTL can exceed observed p99 origin refill latency.

Evidence to collect

• •Distributed traces for the affected key showing miss, lock acquisition, origin call, and cache write events per caller, with caller identity and timestamp deltas.
• •Lock backend logs or audit records (acquire, release, expiry) for the lock key matching the cache key, scoped to the stampede window.
• •Cache backend metrics for key hit, miss, and write counts on the affected key, compared with origin request counts for the same logical identifier.
• •Application configuration snapshot: cache client library and version, lock client library and version, key serialization rules, TTL values, and namespace prefixes.
• •Code coverage or call-graph evidence of which functions call the origin data source, including paths outside the protected cache wrapper.
• •Origin-side metrics (query or RPC counts per identifier) cross-referenced against cache miss counts to quantify the stampede ratio.

Where to look

• •At the cache-aside wrapper boundary: the function that orchestrates miss detection, lock acquisition, origin fetch, and write-back, since this is where bypass typically occurs.
• •At the lock primitive boundary: the key passed to the lock acquire call versus the key passed to the cache get/set call; equality here is the single correctness invariant.
• •At the origin data accessor boundary: search for direct callers of the data access object or repository method that the wrapper exists to protect.
• •At the background job and admin tool boundary: warmup, prefetch, repair, and cache-priming scripts that may write the cache without acquiring the lock.
• •At the configuration loading boundary: environment-specific cache and lock client factories that may produce different prefixes or serializers per process.
• •At the TTL boundary: cache key TTL versus lock key TTL versus measured p99 origin refill time, since short lock TTLs release protection before the refill completes.

Diagnostic steps

1. 01Reconstruct the refill sequence from a representative trace: order the events as miss, lock-acquire-attempt, lock-acquire-result, origin-call-start, origin-call-end, cache-set, lock-release, and confirm the lock-attempt precedes the origin call in every caller.
2. 02Compute, across the stampede window, the ratio of origin calls to cache misses for the affected key; a ratio greater than one confirms concurrent refill.
3. 03Diff the lock key string versus the cache key string in the source code and at runtime via debug logging; any divergence at character level eliminates coordination.
4. 04Enumerate every code path that writes to the cache for this entity and verify each path acquires the lock with the same key before the write; flag any path that does not.
5. 05Compare lock TTL with the tail latency of origin fetches (p95 and p99) for the same entity; lock TTL below p99 refill time indicates ownership expiry mid-refill.
6. 06Verify that the cache backend and lock backend share consistent view semantics: a lock acquired in one store and a cache written in another can race during network partition or replication lag.
7. 07Confirm whether duplicate refills correlate with deployment restarts, connection pool reinitialization, or cache client reconfiguration; mid-flight config changes can disable lock acquisition transiently.

Common mistakes

• •Assuming the presence of a lock-acquiring function in the codebase means it is actually executed on the live refill path; wrappers can be bypassed by direct repository calls during optimization.
• •Using a globally random or instance-scoped suffix on the lock key while using a stable key for the cache, producing N lock namespaces for one cache key.
• •Setting lock TTL based on a guess rather than measured refill latency, which causes owners to lose the lock while followers proceed.
• •Mixing cache and lock clients across different datastores (Redis cache versus database advisory lock, in-process versus distributed lock) without verifying cross-store atomicity.
• •Trusting single-process dedup (in-process mutex) as a stampede defense in a multi-instance deployment, where each replica can independently refill.
• •Treating cache-aside hit rate as sufficient evidence and ignoring origin-side duplicate call counts, which only appear in origin metrics, not cache metrics.

Safe fixes

• •If the lock key string differs from the cache key string, correct the key construction so both are derived from the same canonical identifier and hash; verify with debug logging in a non-production environment first.
• •If a direct origin call exists outside the wrapper, route it through the same cache-aside wrapper or remove the duplicate path; restrict repository-level access to the wrapper where feasible.
• •If lock TTL is shorter than p99 origin refill time, raise lock TTL to a value derived from observed tail latency plus a safety margin, then re-measure duplicate origin counts.
• •If lock and cache live in different stores and partition tolerance is required, adopt a sequence that writes the cache before releasing the lock and uses a fencing or version check to prevent stale overwrites.
• •If multiple cache client instances are in use, consolidate to one configured client with a single key namespace and serializer for the affected entity.
• •If background jobs refill without coordination, either serialize them via the same lock key or exclude the warmup period from origin-load SLAs by pre-seed at controlled cadence.

Prove the fix

1. 01Origin request count for the affected identifier equals the cache miss count for the same identifier across a multi-instance window, with a ratio at or below one.
2. 02For each sampled stampede trigger (TTL expiry, eviction, invalidation), at most one trace shows the origin fetch for that key, and all other contenders show either a blocked wait on the lock or a subsequent cache hit after the owner's write.
3. 03Lock backend metrics show non-zero acquire counts for the lock key during cold-cache windows, and acquire-result returns the exclusive owner for every successful writer.
4. 04The cache key string and the lock key string, captured at runtime for the same logical entity, are byte-equal; a regression test asserts equality across random inputs.
5. 05Lock TTL, evaluated from configuration, exceeds p99 origin refill latency for the entity at the time of change and is revalidated on each release.
6. 06Continuous tracing over a full TTL-expiry cycle shows no origin fetch from any code path other than the lock-winning path, including background jobs and admin tools.

Prevention and next steps

• •Centralize cache-aside access in one wrapper per entity type and forbid direct repository access from application code paths that serve user requests.
• •Add a unit or integration test that asserts cache key and lock key are derived from the same canonical input, and that failing this assertion aborts the build.
• •Publish dashboards that compare origin-to-cache call ratios per key prefix, and alert when the ratio exceeds one plus a configured tolerance.
• •Lock TTL should be calculated from observed origin tail latency with an explicit safety factor, and revalidated whenever the origin dependency changes.
• •Document the stampede-protection contract and require code review sign-off for any new caller of the origin data accessor for a protected entity.

Safe commands and checks

grep -RIn --include=<ext> "<lock_acquire_call_name>" <repo_root>
grep -RIn --include=<ext> -E "(cache_get|cache_set).*<key_pattern>" <repo_root>
grep -RIn --include=<ext> "<repository_or_dao_method>" <repo_root>
grep -RIn --include=<ext> "lock[._-]?ttl" <repo_root>
grep -RIn --include=<ext> "key[._-]?prefix\|namespace" <repo_root>
awk -F'lock_key=' 'NR>1{print $2}' <trace_export_csv> | sort -u | head -n 20