Redis · intermediate

Redis TTL refresh loop: find why stale values never expire

Diagnose and resolve a Redis TTL refresh loop where repeated reads or background refreshes continually extend a key's expiration, preventing intended cache invalidation and leaving stale values alive far longer than the configured TTL.

The symptoms

  • Cache keys persist in Redis memory far longer than the configured TTL, observed via TTL command returning near-maximum values after many minutes or hours.
  • Application consistently reads stale data even after the upstream source-of-truth was updated, despite TTL being set on the cache key.
  • Memory usage in Redis grows without bound for a key namespace expected to be bounded by TTL, visible through INFO memory or per-database key counts.
  • Hot keys show monotonic TTL refresh patterns in slowlog or command traces, with EXPIRE or PEXPIRE called repeatedly against the same key within a short window.
  • Cache miss rate stays near zero even when expected to spike after upstream changes, because reads find the perpetually-refreshed stale key.

Likely causes

  • Read-through or cache-aside client wrapper calls EXPIRE on every successful GET, sliding the TTL forward on each read instead of only on initial SET.
  • Background refresher or "stampede protection" goroutine calls SETEX or EXPIRE on a fixed key after every successful upstream fetch, never allowing natural expiry.
  • Two writers or processes both treat themselves as authoritative and race to refresh TTL after each other, forming an unintended keepalive loop.
  • Key naming scheme collides between users, tenants, or feature flags so that a read for one identity inadvertently touches a TTL on another's key.
  • Lua script or MULTI/EXEC block issues an EXPIRE inside the same transaction as the GET, causing TTL refresh to occur even on read paths.
  • Misread of GETEX semantics: client expects GETEX to leave TTL untouched but actually replaces TTL with a new value, and this happens on every read.

First ten minutes

  1. 01Confirm the symptom shape: pick one suspected "stale" key from the cache namespace and run TTL against it from redis-cli; observe whether the value grows or resets after a known read path executes.
  2. 02Capture a 60-second SLOWLOG or MONITOR trace scoped to that single key and look for repeating EXPIRE, PEXPIRE, SETEX, SET ... EX, or GETEX patterns.
  3. 03Grep the application source for the cache key namespace and identify every code path that issues EXPIRE, PEXPIRE, SETEX, SET ... EX, or GETEX with an expiration option.
  4. 04Disable or feature-flag the suspected refresher goroutine, then re-measure TTL on the key across one full expected expiry window to confirm natural expiration resumes.
  5. 05Verify the cache wrapper's documented contract: is EXPIRE meant to fire only on writes, or on every read? Cross-check against the official cache-aside pattern guidance.

Evidence to collect

  • Per-key TTL samples taken before, during, and after a known upstream update, with timestamps and the operation that immediately preceded each sample.
  • SLOWLOG GET or MONITOR output showing the sequence of commands touching the affected key, including client IP and command arguments.
  • Source-level excerpts of the cache client wrapper showing exactly when EXPIRE or PEXPIRE is invoked and the conditional guard around it.
  • CONFIG GET maxmemory-policy and INFO memory output to confirm whether eviction behavior could be masking or amplifying the symptom.
  • CONFIG GET lazyfree-lazy-expire and active-expire-effort to characterize how aggressively Redis is reclaiming expired keys on this instance.

Where to look

  • At the Redis protocol boundary: SLOWLOG, MONITOR (use sparingly), and the per-command audit trail for repeated EXPIRE-class commands on the same key.
  • Inside the application cache client: the read-through wrapper, the get-or-load helper, and any decorator that adds jitter or stampede protection around cache reads.
  • In the cache key namespace: the formatter that constructs key names, especially around tenant, locale, and feature flag dimensions that could collide.
  • At the background worker boundary: any scheduled refresher, warmup job, or "refresh-ahead" task that touches the cache on a timer.
  • At the deployment topology: multiple replicas of the application issuing concurrent reads or refreshes, particularly behind a load balancer with sticky sessions.

Diagnostic steps

  1. 01Run TTL on the suspected key in steady state, then trigger exactly one upstream update and one known read; observe whether TTL resets upward rather than counting down. This distinguishes a TTL refresh loop from a legitimate fresh write.
  2. 02Capture SLOWLOG GET N or a bounded MONITOR window filtered to the key prefix; if multiple EXPIRE/PEXPIRE calls appear per minute from the same client IP without intervening SET, this is the refresh loop signature.
  3. 03Trace the application code path: instrument the cache wrapper to log every call to EXPIRE/PEXPIRE with the calling function name and a stack sample, then look for a single function dominating the calls.
  4. 04Compare intended vs. actual semantics by reading the client library documentation for SET ... EX, SETEX, GETEX, and EXPIRE; many libraries expose an "always set TTL" mode that surprises engineers.
  5. 05Rule out eviction interference by sampling DBSIZE and used_memory over the affected TTL window; if memory pressure is high and maxmemory-policy is allkeys-lru, expiry may be artificially delayed rather than refreshed.
  6. 06Reproduce in isolation: write a minimal script that performs only GET against the key with no upstream call, and confirm whether TTL still resets; this isolates client behavior from application logic.

Common mistakes

  • Assuming TTL was never set when it was set, but reset on every read; the key "has a TTL" but it is perpetually pushed forward, which looks identical to "no TTL" only over long windows.
  • Concluding the cache wrapper is correct because the unit test passes; tests typically cover SET-then-GET and miss-then-load, not the steady-state "many reads" path where the loop manifests.
  • Adding even more aggressive refresh logic (for example, a stampede-protection refresh that runs every few seconds) in an attempt to "fix" cache freshness, which deepens the loop.
  • Confusing keyspace notifications (notify-keyspace-events) with TTL behavior; the notifications are emitted on natural expiry, which never occurs in a refresh loop and therefore appear silent.
  • Treating the symptom as an eviction problem and tuning maxmemory-policy, when the actual issue is a client that resets TTL and prevents expiry from ever being needed.

Safe fixes

  • If the refresh originates from the read path, remove the EXPIRE/PEXPIRE call from the GET wrapper and ensure TTL is set only in the SET or SETEX branch where the value is first written.
  • If a background refresher is intended, gate it on observed miss rate or last-write timestamp rather than a wall-clock timer, and ensure it uses SETEX (which replaces the value) rather than EXPIRE (which only slides TTL).
  • If GETEX is in use, switch to plain GET so that the existing TTL is preserved on each read, and reserve explicit TTL changes for the write path only.
  • Add a regression guard in the cache wrapper that fails closed if EXPIRE is called on a key that was not just SET by the same code path within a short window, log the violation, and alert.
  • Document the cache contract per key namespace: "TTL set on write only", "TTL refreshed by named refresher at N second cadence", or "TTL never refreshed"; enforce the contract in code review.

Prove the fix

  1. 01After deploying the fix, TTL on the previously-affected key decreases monotonically between writes and reaches -2 (key absent) within the configured TTL window without any intervening EXPIRE-class command in SLOWLOG or MONITOR.
  2. 02SLOWLOG over a 10-minute window shows exactly one EXPIRE/PEXPIRE per cache write, not per cache read, verified by correlating slowlog entries with application access logs.
  3. 03An upstream source-of-truth update becomes visible to readers within one TTL window, demonstrated by a controlled end-to-end test that writes upstream, waits, and asserts the new value is returned.
  4. 04Memory usage for the affected key namespace stabilizes rather than growing, with used_memory and DBSIZE trending in line with traffic rather than exceeding the TTL-bounded expectation.
  5. 05A canary run with the cache wrapper regression guard enabled produces zero TTL-on-read violations over a representative traffic window, confirming the loop is closed.

Prevention and next steps

  • Establish a single canonical cache-aside wrapper per language and forbid ad-hoc SETEX/EXPIRE calls in application code; route all TTL-bearing writes through the wrapper.
  • Add a unit and integration test that asserts TTL strictly decreases between two consecutive GETs without intervening writes, catching regressions before they reach production.
  • Track a "TTL refresh per key" counter in metrics and alert on any non-zero rate for keys tagged as "TTL on write only"; the alert is the earliest signal of a loop forming.
  • Review the cache-aside pattern guidance when adopting a new client library, since some libraries default to refreshing TTL on every successful GET and this default is a common surprise.
  • Periodically audit one production key per namespace by sampling its TTL over a full TTL window; monotonic decay is the proof of correct expiry behavior.

Safe commands and checks

TTL <key>
SLOWLOG GET <count>
CONFIG GET maxmemory-policy
CONFIG GET lazyfree-lazy-expire
CONFIG GET active-expire-effort
INFO memory
DBSIZE
OBJECT HELP