Stale cache entries: prove which write missed invalidation

The symptoms

• •Application readers return a value that no longer matches the authoritative store, and the discrepancy persists across multiple client requests.
• •Refreshing the cache key by hand returns correct data, but subsequent reads drift back to the stale value within seconds.
• •Multiple application nodes serve the same stale value, suggesting the cache is shared rather than the staleness being a process-local artifact.
• •Writes to the source return success, but no corresponding cache evict or set event appears in the cache log or metrics.
• •TTL is long, the value is large, or the key is frequently re-read, which makes the window of staleness visible to users.

Likely causes

• •Write path bypasses the abstraction responsible for invalidation, so the source is updated but the cache key is never touched.
• •Key construction differs between the reader and the write path, so the invalidation hits a different key than the one readers actually fetch.
• •Cache-aside read populates the cache after a write has already invalidated it, racing the invalidation and re-introducing the old value.
• •Invalidation call silently fails or is conditional on a feature flag, error code, or partial-write check that the new code path does not satisfy.
• •Read-through or write-through layer is configured inconsistently across services, so some writes go through the layer and some do not.

First ten minutes

1. 01Capture the exact stale value, the key name, and the source row or document identifier so the comparison is unambiguous.
2. 02Read the current source value from the authoritative store and record both timestamps and any version, etag, or updated_at field.
3. 03Enumerate every code path that writes the source record and list whether each path calls an invalidate, set, or delete on the cache.
4. 04Compare the key construction in the read path against the construction in each write path, including format, case, namespace, and versioning prefix.
5. 05Inspect cache metrics for the key in the relevant window: hits, misses, evictions, and any explicit DEL or SET commands.
6. 06Trace one suspect write end-to-end and confirm whether the cache call was reached, what arguments it used, and whether it returned success.

Evidence to collect

• •The stale value string or payload, the cache key, and the source record identifier, captured at the same clock time.
• •The source record's current value, its last-modified timestamp, and any version, sequence, or hash field the application maintains.
• •Cache server logs or slowlog entries filtered to the suspect key, including DEL, SET, EXPIRE, and any pipeline or script executions.
• •Application logs or traces for the suspected write path, including the cache client call, its result, and any swallowed exception.
• •Cache metrics counters for the key, including hit, miss, eviction, and explicit delete events before and after the suspect write.
• •Configuration snapshot of the cache layer, including TTL, eviction policy, replica count, and any read-through or write-through adapters.

Where to look

• •The boundary between the application write handler and the cache client, where the invalidate or set call is expected to occur.
• •The key construction boundary, where a helper, serializer, or schema version differs between the writer and the reader.
• •The cache server boundary, where DEL, SET, EXPIRE, and script invocations are recorded in the server log or slowlog.
• •The transaction or unit-of-work boundary on the source, where a rollback, retry, or partial commit can decouple source write from cache call.
• •The cache cluster boundary, where replication, failover, or stale replica reads can present a value that no node actually wrote.

Diagnostic steps

1. 01Reproduce the staleness deterministically: run the suspect write, then read the cache key, and record both the stale value and the source value to confirm the gap is real and not a reporting artifact.
2. 02Diff the key strings produced by the read path and each write path, including any hash, suffix, or namespace prefix, and treat any mismatch as a confirmed invalidation miss.
3. 03Replay the suspect write while observing the cache slowlog or command stream to verify whether a DEL, UNLINK, or SET is actually issued and to which key.
4. 04Check for a cache-aside race by introducing a short delay between the write and the invalidation in a test environment, and confirm whether a concurrent read repopulates the stale value.
5. 05Verify the cache client's return value and any error handling on the write path, since a failed invalidate that is caught and ignored will leave the old value untouched.
6. 06Confirm whether the cache cluster is in a replicated or sharded configuration that could serve a stale value from a node that has not yet received the invalidation.
7. 07Correlate the suspect write's timestamp with the cache key's last-modified timestamp and TTL to determine whether the value could simply be older than expected.

Common mistakes

• •Assuming the cache TTL is the cause of staleness and increasing it, when the real cause is that the invalidation never occurred.
• •Adding a global cache flush or pattern delete as a workaround, which masks the missing write path and does not identify which write missed invalidation.
• •Blaming cache replication when the original write path never issued an invalidate, so every replica is consistently wrong.
• •Comparing only the reader's key to the documentation, instead of comparing the reader's key to the writer's key, which is the actual contract that must match.
• •Ignoring silent failures in the cache client, where network errors, timeouts, or serialization errors are caught and the write is reported as successful.

Safe fixes

• •Add an explicit invalidate or set call at the exact boundary where the source write commits, and gate the change behind a feature flag so it can be enabled per service.
• •Centralize key construction in a single helper shared by both readers and writers, and add a unit test that asserts the reader and writer produce the same key string for a given identifier.
• •On the write path, log the cache call's return value or error explicitly rather than swallowing it, so future invalidation failures are visible in logs.
• •Order the operations as write-then-invalidate for the common case, and only consider invalidate-then-write if you have measured a write-skew that actually requires it.
• •Add a version, etag, or hash to the cache value and have readers ignore values whose version is older than the source's current version, as a defensive read-side check.

Prove the fix

1. 01Run the suspect write path in a test environment and verify that a DEL, UNLINK, or SET to the reader's exact key appears in the cache server log within the same request.
2. 02Read the cache key immediately after the write and confirm the value matches the source, and repeat the read several times to rule out a repopulation race.
3. 03Deploy a synthetic check that writes a known source value, reads the cache key, and fails the build if the two diverge beyond an explicit allowed staleness window.
4. 04Add a metric that counts invalidate calls per write path and alerts if the count drops to zero for any code path that historically issued invalidations.
5. 05Add a regression test that exercises every write path and asserts the cached key is absent or refreshed after the write returns, so future refactors cannot reintroduce the gap.

Prevention and next steps

• •Define a single key-construction function per resource and require every reader and writer to import it, with a lint rule or code review check that blocks ad-hoc keys.
• •Wrap source writes and cache invalidations in a single repository or unit-of-work component so it is structurally impossible to commit a write without its invalidation.
• •Record cache invalidation outcomes as structured log fields and dashboard them, so a silent failure in the cache client becomes a visible signal rather than a user report.
• •Review the cache invalidation contract whenever a new write path is added, and require the same evidence checklist used in this guide as part of the change review.

Safe commands and checks

redis-cli -h <host> -p <port> GET <key>
redis-cli -h <host> -p <port> OBJECT IDLETIME <key>
redis-cli -h <host> -p <port> TTL <key>
redis-cli -h <host> -p <port> SLOWLOG GET 50
redis-cli -h <host> -p <port> INFO commandstats | head -n 40
redis-cli -h <host> -p <port> --stat -i 1
redis-cli -h <host> -p <port> MONITOR | head -n 200
redis-cli -h <host> -p <port> CONFIG GET maxmemory-policy