Redis invalidation race: show how an old value returns after a fresh write

The symptoms

• •A user reports seeing a previous value (name, price, role, status flag) immediately after a successful write endpoint returned 200 with the new value.
• •Read endpoints intermittently serve the old payload, but the next request returns the correct value without any further write.
• •Background jobs or scheduled processors see a value that was updated milliseconds earlier on the origin database.
• •Telemetry shows a non-zero but bounded stale-window between the write commit and the cache serving fresh data, often correlated with bursty read traffic on the affected key.
• •Disabling the cache layer eliminates the symptom entirely, but reintroducing it on a single shard reproduces the condition within seconds.

Likely causes

• •Classic cache-aside race: a concurrent reader executes a database miss, then the writer updates and deletes the cache key, but the reader's later database fetch and SET overwrites the cache with the pre-write value.
• •Delayed or batched invalidation: the writer commits the change but enqueues or defers the cache delete, so reads in the interval between commit and invalidation continue to serve the old cached value.
• •Write-through or write-behind path that updates the cache before the database commits, then a transaction rollback leaves the cache holding a value the database never accepted.
• •Replication or read-replica routing where the cache is populated from a replica that has not yet received the write, so the SET captures stale replica state.
• •Key naming or TTL collisions that cause a delete to remove a sibling key (for example due to a hash-tag scope or a shared prefix) while the originally targeted key remains populated.
• •Transactional boundary error: the cache invalidation is performed outside the same transaction as the database write, so a partial failure leaves the cache untouched.

First ten minutes

1. 01Capture the exact request pair: log the write request identifier, the read request identifier, the affected cache key, and wall-clock timestamps at the application layer so the ordering can be reconstructed.
2. 02Confirm the read actually consulted the cache by inspecting the cache-hit/miss metric and the request path; a stale read with a cache-miss indicates a fill race, not a delayed invalidation.
3. 03Compare the write path's invalidation moment to the database commit time using logs and MONITOR or SLOWLOG output on the Redis instance to see whether the DEL arrived before, after, or during the reader's GET.
4. 04Decide which race class you are looking at: if the DEL ran after the reader's GET and before the reader's SET, it is a stale-fill race; if the DEL was not issued at all in the window, it is a delayed invalidation.
5. 05Inspect the writer code path for any deferral, batching, message-queue handoff, or transactional asymmetry that separates the database commit from the cache delete.
6. 06Lock the environment by pinning the affected key with a temporary long TTL or short-circuit invalidation while you gather evidence so further reads do not overwrite the diagnostic state.

Evidence to collect

• •Application logs for the write and read requests including the cache key, the database transaction identifier, and the timestamps of GET, SET, and DEL operations.
• •Redis SLOWLOG or MONITOR trace covering the affected key, showing the relative order of GET, SET, and DEL within the stale window.
• •Cache hit/miss counters and key-level TTL observations for the affected key, sampled before and during the race window.
• •Database commit logs or row-version data proving the write was durable before any reader observed the old value.
• •Code-path evidence showing whether the invalidation is inline with the write transaction, queued, or delegated to a separate worker.

Where to look

• •At the cache-aside boundary: the function or middleware that decides whether to read from cache, hit the origin, and write back to cache.
• •At the write transaction boundary: the point where the database commit completes and where the cache invalidation is issued, looking for any code path between commit and DEL.
• •At the invalidation dispatch boundary: any queue, event bus, scheduled job, or message handler that is responsible for issuing the DEL command.
• •At the key naming boundary: shared prefixes, hash tags, or logical groupings that could cause a DEL aimed at one logical key to touch a sibling, or a SET to land on the wrong slot.
• •At the read-replica boundary: any path where the cache is populated from a follower connection rather than the primary, which can return data older than the cache invalidation event.
• •At the retry boundary: any client-side retry, pipeline, or transaction wrapper that can reorder commands relative to the application's intent.

Diagnostic steps

1. 01Reproduce under controlled concurrency by issuing a write and a flood of reads against a single key while logging cache command order; if the race is observable, capture the exact ordering for evidence.
2. 02For each candidate cause, add a discriminating instrument: a request-scoped marker that travels with the read so you can tell whether a stale fill came from the same logical request that observed the miss.
3. 03Distinguish stale-fill from delayed-invalidation by toggling the writer between inline DEL and DEL-after-commit-on-shutdown; only delayed-invalidation should respond to the inline path.
4. 04Check replication lag if reads are served from a follower and the cache is filled from that follower; a non-zero lag combined with cache fill on the follower is sufficient to explain the symptom without any application race.
5. 05Audit key naming for collision risk by listing all keys that share a hash tag or prefix with the affected key and verifying that DEL commands reach exactly one logical entry.
6. 06Verify transactional coupling by checking whether the database commit and the cache invalidation share the same logical unit; if not, you have identified a structural delayed-invalidation risk even when not currently triggered.
7. 07Re-run the reproduction after each instrumentation change so the cause is narrowed by elimination rather than assumption.

Common mistakes

• •Treating the symptom as a cache TTL problem and increasing TTL, which can lengthen the stale window rather than eliminate the race.
• •Adding a second DEL without addressing the read-after-miss SET, which leaves the stale-fill race intact because the read can still SET the old value after the new DEL.
• •Moving invalidation to a background queue without ordering guarantees, which converts a synchronous race into an asynchronous one and makes the failure harder to reproduce.
• •Assuming the cache library is atomic; cache-aside reads are not atomic with respect to a concurrent writer unless the application enforces ordering.
• •Populating the cache from a read replica without checking replication lag, so the cache is filled with data older than the latest write.
• •Key naming collisions caused by shared prefixes or hash tags that make a DEL command ambiguous about which logical entity is being invalidated.

Safe fixes

• •If the evidence shows a stale-fill race (the DEL arrives between the reader's miss and its SET), switch the read path to a single-flight or request-scoped lock so only one reader can populate the cache for a given key within a stale window, scoped to the specific endpoint and key.
• •If the evidence shows a delayed invalidation, move the DEL to be issued inline before the write transaction commits a response, or use a transactional outbox so the DEL is guaranteed to follow a successful commit, scoped to the affected writer path.
• •If the cache is populated from a read replica with non-trivial lag, restrict cache-fill reads to the primary for the affected key, or skip the cache entirely until lag is below a documented threshold.
• •If key naming collisions exist, introduce a versioned key or include a discriminator in the key so that DEL targets exactly one logical entry, scoped to the affected key namespace.
• •If the write path uses a queue or worker for invalidation, add an ordering constraint keyed on the database transaction or write timestamp so out-of-order invalidations cannot reintroduce stale data.

Prove the fix

1. 01Run the controlled reproduction again and confirm via application logs and Redis command traces that for the affected key, every read following a write observes the post-write value within the documented stale window.
2. 02Verify the absence of late SETs that overwrite a fresh DEL by inspecting command ordering traces during the reproduction: a successful fix shows no SET of the pre-write value after the post-write DEL.
3. 03Confirm the fix is local to the affected code path by toggling the change off and on while running the reproduction; the symptom should track the toggle exactly.
4. 04Observe cache hit/miss and stale-read metrics over a fixed observation window after deployment and confirm the stale-read rate for the affected key returns to the documented baseline.
5. 05Add a regression test that issues a write followed by N concurrent reads against the same key and asserts that all reads return the post-write value, so the failure mode cannot return silently.

Prevention and next steps

• •Document the cache-aside contract for each endpoint: where the cache is read, where it is filled, where it is invalidated, and what ordering guarantees exist between the database commit and the DEL.
• •Use versioned or scoped cache keys for any entity that can be written concurrently with reads, so collisions cannot cause cross-entity invalidation.
• •Centralize cache invalidation in a single helper that takes the write transaction result and issues the DEL inline, rather than scattering DEL calls across request paths and workers.
• •Instrument request-scoped markers on read paths so stale fills can be attributed to the originating request during incident analysis.
• •Review cache-aside code paths in code review with a checklist that asks explicitly whether the read-after-miss SET can race a concurrent writer's DEL.

Safe commands and checks

redis-cli MONITOR | grep -E 'GET|SET|DEL' filters Redis traffic to the commands that participate in a cache-aside race so you can reconstruct ordering without changing state.
redis-cli SLOWLOG GET 50 inspects recent slow commands on the Redis instance to identify long GET or SET operations that may indicate lock contention during a stale window.
redis-cli OBJECT IDLETIME <key> reports how long the affected key has been idle, helping confirm whether a long TTL is masking the race rather than preventing it.
redis-cli --scan --pattern '<key-prefix>*' enumerates keys sharing a prefix with the affected key so you can detect hash-tag or prefix collisions that cause DEL ambiguity.
redis-cli INFO replication surfaces role and connected replicas so you can confirm whether the cache-fill path can reach a follower with non-zero replication lag.
redis-cli CLIENT LIST shows connected clients and their command subscriptions, useful for confirming that an invalidation worker is currently attached when diagnosing delayed-invalidation cases.