Cache consistency debugging checklist

The symptoms

• •Reader A receives a value that disagrees with a value retrieved by Reader B for the same key within the cache's stated lifetime.
• •A write to the origin returns success, yet a subsequent read from the cache layer continues to serve the pre-write value past the configured TTL.
• •Different geographic regions or application instances return different values for the same cache key, indicating partition or replication drift.
• •Logged Cache-Control directives on the response do not match the directives set by the application or gateway configuration.
• •Stale values appear only after a deployment, configuration reload, or cache-node restart, suggesting warm-up or cold-cache inconsistency.

Likely causes

• •Time-to-live (max-age or s-maxage) is set longer than the acceptable staleness window, so reads outpace invalidation.
• •Cache-Control contains must-revalidate or no-cache directives that the cache layer is ignoring, allowing stale reuse without re-checking the origin.
• •Write-through invalidation is missing or asynchronous, so a successful origin write is not propagated to every cache node before the next read.
• •Key construction includes a variable (user id, region, version stamp) that omits a dimension, causing two semantically distinct reads to collide on one cache entry.
• •A reverse proxy, CDN edge, or browser intermediate cache retains a copy governed by a Cache-Control header that the application author did not intend for that tier.

First ten minutes

1. 01Capture the exact symptom: which key, which reader path, what value was expected versus observed, and the timestamp of each observation.
2. 02Identify the cache tier involved (in-process, shared, reverse proxy, CDN edge) by inspecting the response headers and the deployment topology.
3. 03Record the Cache-Control directives returned by the origin for the affected URL and compare them to the policy the application intends.
4. 04Determine whether the failure reproduces after a forced cache eviction, to distinguish between a too-long TTL and a missing invalidation path.
5. 05Form a falsifiable hypothesis: e.g., "Reader B sees a stale value because node N did not receive the invalidation message issued at time T."

Evidence to collect

• •Response headers for the affected URL, focusing on Cache-Control, Age, ETag, Last-Modified, Vary, and Date.
• •The Age header value compared against the configured max-age or s-maxage, to detect whether a cache entry is older than allowed.
• •Application or cache logs showing cache HIT versus MISS versus STALE events for the affected key, with timestamps and node identifiers.
• •The invalidation log entry (channel, topic, or message ID) for the write that should have invalidated the cache entry.
• •Configuration values for TTL, invalidation mode (write-through, write-behind, time-based), and key-construction function in the active deployment.

Where to look

• •The boundary between the application process and the cache layer: the cache client wrapper, its serialization, and its key-construction function.
• •The boundary between the origin server and any reverse proxy or CDN: the Cache-Control header emitted by the origin versus what the edge returns.
• •The boundary between write paths and cache invalidation: the write handler's commit step and the message that triggers cache eviction or refresh.
• •The boundary between cache nodes in a cluster: the replication or gossip channel that propagates invalidations across instances.
• •The boundary between HTTP semantics and the cache layer: directives such as must-revalidate, no-cache, private, and s-maxage and which tier honors them.

Diagnostic steps

1. 01Compare the value returned by the cache tier to the value returned by the origin for the same key, and record the Cache-Control and Age headers on each path.
2. 02Replay a known write to the origin and observe whether the cache returns the new value immediately, after one TTL, or never within the TTL window.
3. 03For each cache node serving the key, check whether an invalidation event for that key has been processed, using the cache's own audit log or metrics.
4. 04Inspect the key-construction function to confirm it includes every dimension required to distinguish the observed variants (for example, tenant, region, schema version).
5. 05Verify that the Cache-Control directives emitted by the origin match the directive the cache layer is expected to honor, consulting the HTTP Cache-Control specification for each directive's semantics.
6. 06If multiple tiers are involved, test each tier in isolation by bypassing the upstream tier, to localize which boundary retains the stale value.

Common mistakes

• •Assuming "stale" means "TTL too long" without checking whether invalidation was actually attempted, sent, and received by every serving node.
• •Trusting a cache HIT log line as proof of freshness, when the underlying entry may be past its configured lifetime but not yet evicted.
• •Setting Cache-Control on the application response but not on the reverse proxy or CDN configuration, so the edge tier governs the entry instead.
• •Constructing cache keys from a partial set of inputs, causing semantically different reads to share one entry and produce apparent inconsistency.
• •Confusing client-side browser caching with server-side cache invalidation; the two require separate evidence and separate fixes.

Safe fixes

• •If the Age header on a returned response exceeds the configured max-age or s-maxage, shorten the TTL to a value inside the staleness budget and re-test.
• •If invalidations are issued but not observed on every node, switch from best-effort broadcast to a durable invalidation channel with per-node acknowledgment, and confirm via audit logs.
• •If two distinct reads collide on one cache entry, extend the key-construction function to include the missing dimension and verify the two reads now produce different keys.
• •If a reverse proxy or CDN ignores must-revalidate or no-cache, set the Cache-Control directives at the edge tier and confirm the origin response carries the same directives for client hops.
• •If a cache tier cannot honor must-revalidate, replace it with explicit short TTL plus synchronous invalidation rather than relying on the directive's semantics alone.

Prove the fix

1. 01Issue a write to the origin and observe that every cache node serving the key returns the new value on the next read, with no STALE log events for that key.
2. 02Compare the Cache-Control directives on the response against the documented policy, and confirm Age does not exceed max-age or s-maxage on any observed read.
3. 03Run a scripted probe that reads the key from N distinct cache nodes and asserts all N return identical bytes within an agreed staleness window after a write.
4. 04After a forced restart of one cache node, observe that the warmed entry it serves matches the current origin value rather than a pre-restart snapshot.
5. 05Add a regression assertion that fails the build if any HTTP response for the affected URL omits the required Cache-Control directive or carries an Age value above the configured maximum.

Prevention and next steps

• •Define an explicit staleness budget per cache tier and pin TTL, invalidation mode, and key dimensions in configuration reviewed with every change.
• •Treat Cache-Control directives as a contract: assert in tests that the origin emits the agreed directives and that each intermediate tier honors them per the HTTP specification.
• •Use synchronous write-through invalidation for keys whose inconsistency cost is high, and reserve TTL-only strategies for keys that tolerate the configured staleness.
• •Instrument every cache tier with HIT, MISS, STALE, and INVALIDATION counters, and alert on STALE events for keys flagged as consistency-critical.
• •Document the key-construction function alongside the data model so reviewers can spot a missing dimension before it ships.

Safe commands and checks

curl -sSI -H 'Cache-Control: no-cache' <origin-url> | grep -iE 'cache-control|age|etag|last-modified|vary|date'
redis-cli -h <cache-host> -p <port> DEBUG OBJECT <key> 2>/dev/null | grep -iE 'age|encoding|lru'
redis-cli -h <cache-host> -p <port> SLOWLOG GET 64 | grep -iE 'evicted|invalidat|expired'
memcached-tool <cache-host>:<port> stats | grep -iE 'curr_items|get_hits|get_misses|evictions'
varnishadm -S <secret-file> -T <admin-addr>:<admin-port> ban 'req.url ~ <path-pattern>'
varnishlog -q 'RespHeader:Cache-Control' -i TxHeader 2>/dev/null
grep -nE 'cache_(get|set|delete)|invalidat' <app-log-path> | grep -i '<key-pattern>'
httpx -H 'Cache-Control: no-cache' -m <origin-url> -print hb 2>/dev/null | grep -iE 'cache-control|age'