How to reproduce and verify cache invalidation

The symptoms

• •Read-after-write returns the previous value on at least one reader path, even though the write was acknowledged by the origin store and the cache client reported success.
• •Two clients reading the same key within the invalidation window observe different values, indicating split visibility rather than latency.
• •Stale values reappear after a key TTL would have expired, suggesting a second cache tier or a copy-on-rebuild path is restoring the old payload.
• •Stale values reappear only on specific endpoints, routes, or serializers, indicating the write path invalidated the wrong key, key family, or representation.
• •Cache hit rate remains high during a window when stale data has been reported, indicating reads are still served from the old representation.

Likely causes

• •The write path mutates the origin store but uses a key name, version suffix, or namespace tag that the read path does not query, so the delete never matches a live entry.
• •Multiple cache layers (edge, application, object cache, query cache) are present and only one is invalidated; readers continue to hit an uninstrumented tier.
• •Cache-aside reads repopulate from the origin before the invalidation propagates, or a write-through populates a new value at a different key than the read path resolves.
• •The invalidated key holds a serialized representation that another reader deserializes through a different schema or codec, so the cache contains a logically different entry under a matching name.
• •Cluster topology changes (slot moves, replica promotion, key migration) cause the invalidation command to land on a node that no longer owns the key after the topology event.

First ten minutes

1. 01Confirm the scope of the failure: identify the exact key, key family, or query that the user reports as stale, and the reader endpoint that surfaces the stale value.
2. 02Determine the read path that serves that endpoint: name the cache layer, the key derivation function, and the fallback to the origin store when the cache misses.
3. 03Determine the write path that should invalidate it: name the operation, the cache client call used for invalidation, and the key derivation function used by that call.
4. 04Compare the two key derivation functions byte-for-byte (prefix, delimiter, version tag, hash input) to decide whether the invalidation can possibly address the entry the reader resolves.
5. 05Enumerate every cache layer between the reader and the origin store, including edge caches, in-process caches, and second-tier caches, and mark which are touched by the write path.
6. 06Reproduce the stale read once against a controlled key, using a known good value before and a known different value after, and record whether the stale read returns the before or after value.

Evidence to collect

• •For the failing key, capture the exact cache key string produced by the read path and the exact cache key string produced by the write path's invalidation call.
• •Capture the timestamp and origin-store response of the write, the timestamp of the invalidation command, and the timestamp of the stale read.
• •Capture the cache client's reported return value for the invalidation command (number of keys removed, queue length, replication acknowledgement) and any client-side error.
• •Capture the cache server's view: key existence, TTL, value length, and the last access time as reported by a read-only inspection command on the failing key.
• •Capture a read-only cache trace showing which layer served the stale response (HIT, MISS, BYPASS) and the originating layer identifier.

Where to look

• •At the boundary between the write transaction commit and the cache invalidation call, where a thrown exception or early return can leave the origin updated and the cache stale.
• •At the boundary between the application cache client and the cache server, where command framing, key encoding, and connection ownership can change the resolved key.
• •At the boundary between the read path's key resolver and the write path's invalidation key resolver, where divergent prefix, suffix, or hash inputs produce different key strings.
• •At each additional cache tier between the application and the origin store, where a tier not covered by the invalidation call continues to serve the old value.
• •At the cluster topology boundary after a failover, slot migration, or resharding event, where the invalidation command may be routed to a node that no longer owns the key.

Diagnostic steps

1. 01Pin a single key with a known sentinel value, perform a read, and verify the cache reports HIT with that sentinel, establishing a baseline read path.
2. 02Perform the production write operation that should invalidate the key, capture the cache client's return value, and immediately read the key through the same read path the user reported.
3. 03Use a read-only inspection of the cache to confirm whether the key still exists; if it exists, compare its serialized form to the pre-write value and the post-write value to identify which representation is reader-visible.
4. 04If the key no longer exists, perform a read that should repopulate it (cache-aside GET) and inspect again to determine whether the repopulated value is the pre-write or post-write value.
5. 05Repeat the read from a second process, container, or connection to rule out a stale in-process cache or connection-bound state rather than a server-side invalidation defect.
6. 06For each additional cache tier identified in the inventory, repeat the read through the read path that traverses that tier and confirm whether the stale value is served from that specific tier.
7. 07If topology is suspect, inspect cluster slot ownership for the key before and after the write to determine whether the invalidation landed on the owning node.

Common mistakes

• •Assuming the cache client return value of an invalidation call proves the entry was removed from every reader-visible cache, when in fact multiple tiers or representations may remain.
• •Comparing key names by visual prefix only, missing a version suffix, tag, namespace, or hash input that makes the invalidation key resolve to a different entry than the reader key.
• •Testing invalidation only on the same connection that performed the write, allowing an in-process or connection-local cache to mask a server-side invalidation defect.
• •Relying on TTL expiry as a substitute for invalidation, which only bounds staleness and does not prove that a write replaced the reader-visible representation.
• •Reproducing the bug against a synthetic key that the production read path never resolves, producing a false negative for the invalidation contract.

Safe fixes

• •If the captured key strings diverge, align the read path's key resolver and the write path's invalidation key resolver to a single derivation function, and re-run the write-then-read protocol to confirm the reader-visible value updates.
• •If a second cache tier is serving the stale value, extend the invalidation call to that tier using its native invalidation API, and verify with a read through the read path that traverses that tier.
• •If cache-aside repopulation races the invalidation, change the invalidation to a delete-and-double-check pattern or move the delete to occur after the origin commit is durable, and verify by repeating the read immediately after the write.
• •If topology changes redirect the invalidation, re-issue the invalidation against the node that currently owns the key's slot, or use a topology-aware client call that addresses the owning node, and verify by inspecting the key after the call.
• •If the bug only manifests under serialization differences, invalidate by a stable identifier (primary key plus version) rather than by serialized payload hash, and verify by reading both representations after the write.

Prove the fix

1. 01A controlled write of a sentinel new value to a pinned key is followed by N reads across all enumerated reader paths; every read returns the new value or a cache miss that subsequently returns the new value, with zero reads returning the pre-write value.
2. 02A read-only inspection of every enumerated cache tier shows no entry whose serialized form equals the pre-write value for the pinned key, and shows no entry under any key string that the read path resolves for the same logical record.
3. 03The same write-then-read protocol is executed from a second process, container, or connection, and produces the same outcome, demonstrating that the fix is not local to one in-process or connection-local cache.
4. 04The protocol is executed a second time after a cluster topology event (failover, slot move, or migration) and continues to produce the same outcome, demonstrating topology-aware invalidation.
5. 05The protocol is executed at least once for each reader path and each cache tier identified in the inventory, with a recorded pass for each combination, providing a regression check that can be re-run on any future change.

Prevention and next steps

• •Treat the key derivation function as a shared module imported by both the read path and the write path, so a change to key shape cannot diverge between the two paths.
• •Maintain a written inventory of every cache tier between the reader and the origin store, and require that every write operation declare which tiers it invalidates and through which key strings.
• •Add a regression test that performs the write-then-read protocol across all reader paths and all cache tiers, and run it on every change that touches the write path, the read path, or the cache layer.
• •Avoid TTL as the sole defense against staleness; require an explicit invalidation call for any write that must be reader-visible immediately, and verify the call in the same change that introduces the write.
• •When introducing topology changes, re-run the invalidation regression suite and confirm the invalidation client call is topology-aware before merging.

Safe commands and checks

Set the sentinel value on a pinned key and verify the cache holds it: SET <pinned-key> <sentinel-before>; OBJECT ENCODING <pinned-key>; TTL <pinned-key>
Read the pinned key through the production read path to confirm a HIT, and capture the cache client's HIT or MISS response code as reported by the client library.
Perform the production write operation that should invalidate the pinned key, and capture the return value of the cache client's invalidation call (for example, an integer count of keys removed) and any client-side error.
Inspect the pinned key with read-only commands to determine whether it exists and what it contains: EXISTS <pinned-key>; STRLEN <pinned-key>; OBJECT IDLETIME <pinned-key>
Read the pinned key through the production read path immediately after the write; record whether the response equals <sentinel-before>, equals the post-write value, or is a miss that repopulates to the post-write value.
For each additional cache tier identified in the inventory, repeat the read through the read path that traverses that tier and record whether the response equals <sentinel-before> or the post-write value; use the tier's native read-only inspection command (such as a keys-by-prefix listing restricted to a test-only namespace) to confirm.
For topology-aware verification, capture the slot for the pinned key before and after the invalidation using the cluster's read-only slot-mapping command, and confirm the invalidation was directed at the node that owns the slot at the time of the call.