Redis · intermediate
Redis TTL refresh loop: find why stale values never expire
Diagnose and resolve a Redis TTL refresh loop where repeated reads or background refreshes continually extend a key's expiration, preventing intended cache invalidation and leaving stale values alive far longer than the configured TTL.
The symptoms
- •Cache keys persist in Redis memory far longer than the configured TTL, observed via TTL command returning near-maximum values after many minutes or hours.
- •Application consistently reads stale data even after the upstream source-of-truth was updated, despite TTL being set on the cache key.
- •Memory usage in Redis grows without bound for a key namespace expected to be bounded by TTL, visible through INFO memory or per-database key counts.
- •Hot keys show monotonic TTL refresh patterns in slowlog or command traces, with EXPIRE or PEXPIRE called repeatedly against the same key within a short window.
- •Cache miss rate stays near zero even when expected to spike after upstream changes, because reads find the perpetually-refreshed stale key.
Likely causes
- •Read-through or cache-aside client wrapper calls EXPIRE on every successful GET, sliding the TTL forward on each read instead of only on initial SET.
- •Background refresher or "stampede protection" goroutine calls SETEX or EXPIRE on a fixed key after every successful upstream fetch, never allowing natural expiry.
- •Two writers or processes both treat themselves as authoritative and race to refresh TTL after each other, forming an unintended keepalive loop.
- •Key naming scheme collides between users, tenants, or feature flags so that a read for one identity inadvertently touches a TTL on another's key.
- •Lua script or MULTI/EXEC block issues an EXPIRE inside the same transaction as the GET, causing TTL refresh to occur even on read paths.
- •Misread of GETEX semantics: client expects GETEX to leave TTL untouched but actually replaces TTL with a new value, and this happens on every read.
First ten minutes
- 01Confirm the symptom shape: pick one suspected "stale" key from the cache namespace and run TTL against it from redis-cli; observe whether the value grows or resets after a known read path executes.
- 02Capture a 60-second SLOWLOG or MONITOR trace scoped to that single key and look for repeating EXPIRE, PEXPIRE, SETEX, SET ... EX, or GETEX patterns.
- 03Grep the application source for the cache key namespace and identify every code path that issues EXPIRE, PEXPIRE, SETEX, SET ... EX, or GETEX with an expiration option.
- 04Disable or feature-flag the suspected refresher goroutine, then re-measure TTL on the key across one full expected expiry window to confirm natural expiration resumes.
- 05Verify the cache wrapper's documented contract: is EXPIRE meant to fire only on writes, or on every read? Cross-check against the official cache-aside pattern guidance.
Evidence to collect
- •Per-key TTL samples taken before, during, and after a known upstream update, with timestamps and the operation that immediately preceded each sample.
- •SLOWLOG GET or MONITOR output showing the sequence of commands touching the affected key, including client IP and command arguments.
- •Source-level excerpts of the cache client wrapper showing exactly when EXPIRE or PEXPIRE is invoked and the conditional guard around it.
- •CONFIG GET maxmemory-policy and INFO memory output to confirm whether eviction behavior could be masking or amplifying the symptom.
- •CONFIG GET lazyfree-lazy-expire and active-expire-effort to characterize how aggressively Redis is reclaiming expired keys on this instance.
Where to look
- •At the Redis protocol boundary: SLOWLOG, MONITOR (use sparingly), and the per-command audit trail for repeated EXPIRE-class commands on the same key.
- •Inside the application cache client: the read-through wrapper, the get-or-load helper, and any decorator that adds jitter or stampede protection around cache reads.
- •In the cache key namespace: the formatter that constructs key names, especially around tenant, locale, and feature flag dimensions that could collide.
- •At the background worker boundary: any scheduled refresher, warmup job, or "refresh-ahead" task that touches the cache on a timer.
- •At the deployment topology: multiple replicas of the application issuing concurrent reads or refreshes, particularly behind a load balancer with sticky sessions.
Diagnostic steps
- 01Run TTL on the suspected key in steady state, then trigger exactly one upstream update and one known read; observe whether TTL resets upward rather than counting down. This distinguishes a TTL refresh loop from a legitimate fresh write.
- 02Capture SLOWLOG GET N or a bounded MONITOR window filtered to the key prefix; if multiple EXPIRE/PEXPIRE calls appear per minute from the same client IP without intervening SET, this is the refresh loop signature.
- 03Trace the application code path: instrument the cache wrapper to log every call to EXPIRE/PEXPIRE with the calling function name and a stack sample, then look for a single function dominating the calls.
- 04Compare intended vs. actual semantics by reading the client library documentation for SET ... EX, SETEX, GETEX, and EXPIRE; many libraries expose an "always set TTL" mode that surprises engineers.
- 05Rule out eviction interference by sampling DBSIZE and used_memory over the affected TTL window; if memory pressure is high and maxmemory-policy is allkeys-lru, expiry may be artificially delayed rather than refreshed.
- 06Reproduce in isolation: write a minimal script that performs only GET against the key with no upstream call, and confirm whether TTL still resets; this isolates client behavior from application logic.
Common mistakes
- •Assuming TTL was never set when it was set, but reset on every read; the key "has a TTL" but it is perpetually pushed forward, which looks identical to "no TTL" only over long windows.
- •Concluding the cache wrapper is correct because the unit test passes; tests typically cover SET-then-GET and miss-then-load, not the steady-state "many reads" path where the loop manifests.
- •Adding even more aggressive refresh logic (for example, a stampede-protection refresh that runs every few seconds) in an attempt to "fix" cache freshness, which deepens the loop.
- •Confusing keyspace notifications (notify-keyspace-events) with TTL behavior; the notifications are emitted on natural expiry, which never occurs in a refresh loop and therefore appear silent.
- •Treating the symptom as an eviction problem and tuning maxmemory-policy, when the actual issue is a client that resets TTL and prevents expiry from ever being needed.
Safe fixes
- •If the refresh originates from the read path, remove the EXPIRE/PEXPIRE call from the GET wrapper and ensure TTL is set only in the SET or SETEX branch where the value is first written.
- •If a background refresher is intended, gate it on observed miss rate or last-write timestamp rather than a wall-clock timer, and ensure it uses SETEX (which replaces the value) rather than EXPIRE (which only slides TTL).
- •If GETEX is in use, switch to plain GET so that the existing TTL is preserved on each read, and reserve explicit TTL changes for the write path only.
- •Add a regression guard in the cache wrapper that fails closed if EXPIRE is called on a key that was not just SET by the same code path within a short window, log the violation, and alert.
- •Document the cache contract per key namespace: "TTL set on write only", "TTL refreshed by named refresher at N second cadence", or "TTL never refreshed"; enforce the contract in code review.
Prove the fix
- 01After deploying the fix, TTL on the previously-affected key decreases monotonically between writes and reaches -2 (key absent) within the configured TTL window without any intervening EXPIRE-class command in SLOWLOG or MONITOR.
- 02SLOWLOG over a 10-minute window shows exactly one EXPIRE/PEXPIRE per cache write, not per cache read, verified by correlating slowlog entries with application access logs.
- 03An upstream source-of-truth update becomes visible to readers within one TTL window, demonstrated by a controlled end-to-end test that writes upstream, waits, and asserts the new value is returned.
- 04Memory usage for the affected key namespace stabilizes rather than growing, with used_memory and DBSIZE trending in line with traffic rather than exceeding the TTL-bounded expectation.
- 05A canary run with the cache wrapper regression guard enabled produces zero TTL-on-read violations over a representative traffic window, confirming the loop is closed.
Prevention and next steps
- •Establish a single canonical cache-aside wrapper per language and forbid ad-hoc SETEX/EXPIRE calls in application code; route all TTL-bearing writes through the wrapper.
- •Add a unit and integration test that asserts TTL strictly decreases between two consecutive GETs without intervening writes, catching regressions before they reach production.
- •Track a "TTL refresh per key" counter in metrics and alert on any non-zero rate for keys tagged as "TTL on write only"; the alert is the earliest signal of a loop forming.
- •Review the cache-aside pattern guidance when adopting a new client library, since some libraries default to refreshing TTL on every successful GET and this default is a common surprise.
- •Periodically audit one production key per namespace by sampling its TTL over a full TTL window; monotonic decay is the proof of correct expiry behavior.
Safe commands and checks
TTL <key> SLOWLOG GET <count> CONFIG GET maxmemory-policy CONFIG GET lazyfree-lazy-expire CONFIG GET active-expire-effort INFO memory DBSIZE OBJECT HELP