Redis cache-key drift: find readers and writers using different contracts

The symptoms

• •Repeated Redis MISS for a resource that was clearly written moments earlier, visible as a hit rate near zero for a specific key family while siblings stay healthy.
• •Latency spikes correlate with cache MISS events for warm logical resources, with origin fetches re-populating the cache in the same request path.
• •Two application instances serve different values for the same id, but only after a deployment or library change; values stabilize per-instance but disagree across instances.
• •Sudden growth in a key family that has no matching reads, or reads that return payload-sized blobs decodable into scalar types where objects are expected.
• •Bloom-filter or guard checks on the read side disagree with the condition the writer used, suggesting the writer computed a different identity than the reader checked.
• •Keys observed in Redis differ from the keys your instrumentation logs, and the divergence starts at a specific commit or rollout boundary.

Likely causes

• •Independent key-construction helpers in different modules, each with its own serialization for ids, scopes, or version tags.
• •A serializer change that alters the canonical form of composite keys, such as adding locale, tenant scope, or hash algorithm, applied to writers but not all readers.
• •Hand-edited key strings in one path versus a shared helper in another, often introduced during a hotfix that bypassed the helper.
• •Prefix drift when a service-name, tenant-id, or region variable differs between writer and reader due to environment configuration.
• •Hashing inconsistency where one side hashes a normalized string and another hashes the raw user input, producing different keys for the same logical resource.
• •Key rotation or namespace migration that was rolled out to some callers but not others, leaving two parallel key families alive.

First ten minutes

1. 01Capture the exact key the writer stored and the exact key the reader looked up for the same logical id; do not rely on application logs alone.
2. 02Run an in-Redis SCAN over the suspected key family and list the actual key strings so you can compare them to your helper output.
3. 03Identify each caller that performs GET, SET, DEL, and EXISTS for the family and tag it with its source file, deploy, and config.
4. 04Inspect Redis SLOWLOG or METRICS for the family's hit ratio and TTL distribution to confirm reads are missing, not failing.
5. 05Compare the two key strings byte by byte, including any case, separator, or trailing-slash differences, before assuming a logical bug.
6. 06Quarantine the suspected family by moving tests to a separate key prefix and stop further writes until the contract is confirmed.

Evidence to collect

• •Side-by-side capture of the writer's key string and the reader's key string for the same logical id, including length and a hex of the raw bytes.
• •Annotated SCAN output listing unique key prefixes and key counts per prefix inside the affected family.
• •List of every code path that calls GET, SET, DEL, SETEX, or EXISTS for the family, with file, function, and config it reads.
• •Hit ratio, miss ratio, and TTL histogram for the affected family from Redis INFO stats or an external metrics source.
• •Deployment and configuration diff between the last known-good and the first observed drift, narrowed to the key-construction area.
• •Two distinct responses from two instances for the same id, kept as a single artifact with timestamps and request inputs.

Where to look

• •Inside Redis, at the boundary between SCAN, INFO keyspace, and TTL output, where the actual keyspace truth lives apart from any application log.
• •At the helper-versus-callsite boundary in the codebase, where one module hands a key string to another and the contract can be dropped.
• •At the configuration boundary between services, where prefix variables are read from environment, config files, or feature flags.
• •At the serialization boundary inside the cache layer, where ids, scopes, and version tags are concatenated or hashed before storage.
• •At the deploy and rollback boundary, where multiple versions of the same binary run concurrently and can disagree on the helper.
• •At the migration boundary, where a previous key family was renamed and old keys may still be served by some readers.

Diagnostic steps

1. 01Use SCAN with the family's pattern and group the resulting keys by their prefix and tail to expose silent duplicates that survive a naive helper.
2. 02Compose the writer key and the reader key for the same id in a controlled test and diff them bytewise, including any case and separator, before concluding the code is wrong.
3. 03Trace every callsite that constructs a key for the family and list each at a single table with its source file, function, and the helpers it uses.
4. 04Compare the helper outputs against the actual SCAN output to see whether the visible keys match what the helper would produce, which is the load-bearing evidence for drift.
5. 05Cross-check the metric hit ratio for the family against the application's logged miss rate; if Redis shows hits but the app reports misses, the read key is wrong.
6. 06Reproduce a single read with command-line tooling to confirm the failure with a fixed input, then trace the values through the helper to localize the divergence.
7. 07Decide between realigning the helper for all callers and introducing a versioned key prefix based on whether the old key family must remain readable during migration.

Common mistakes

• •Reading from cache and never noticing the miss because the origin serves a fresh value, so the bug presents as latency rather than correctness.
• •Trusting application-side keys without verifying against SCAN, which lets typo or empty-string prefixes hide in the helper layer.
• •Assuming a library is uniform when two versions of the same library are loaded, each producing a different key shape.
• •Fixing one callsite and missing the others because the helper itself is the source of truth, leaving drift to recur in the next caller.
• •Rotating a key prefix without a dual-read strategy, so readers that still send the old key family miss for the entire migration window.
• •Confusing encoding differences with drift, for example comparing a UTF-8 string to a JSON-escaped string that represents the same resource.

Safe fixes

• •Centralize all key construction for the family in a single helper and route every callsite through it, then re-run the SCAN comparison to confirm parity.
• •If a config-driven prefix is the source of drift, document the exact constructor at the boundary and add a startup self-check that fails fast when the prefix is empty.
• •When the serializer changed, freeze the helper at the old behavior for existing families and add a new helper version for new families, with parallel reads until the old family ages out.
• •Use a versioned prefix such as a single character schema tag so a future change can be rolled out without invalidating the entire cache at deploy time.
• •Add a read-side fallback that tries the alternative key under a feature flag, then compare values to confirm the contract before turning the fallback into the primary path.
• •Wrap the helper with a canary that emits both the new and old key for a sample of traffic, then validate equality before the wider rollout.

Prove the fix

1. 01For a fixed input id, the writer's stored key and the reader's lookup key are byte-equal when inspected through SCAN and a manual probe, with no parallel key families outside the versioned prefix.
2. 02Hit ratio for the family returns to the pre-incident level within a configurable window after the helper change is fully rolled out, with no new key prefixes appearing in SCAN.
3. 03Two application instances served the same id at the same time and returned identical values, with the divergence timestamp logged and absent after the change.
4. 04Application-side miss rate for the family matches the Redis miss rate within noise, so writers and readers are observing the same keyspace.
5. 05A regression test exists that runs the writer and reader helpers against the same id and asserts byte-equal output, and the test is wired into the deployment pipeline.
6. 06No untouched keys in the legacy family remain after the agreed TTL window, confirmed by a final SCAN that returns zero matches for the old pattern.

Prevention and next steps

• •Make the key constructor a single function with a documented contract, and forbid string concatenation outside it for any cache key.
• •Add a contract test that asserts equality between the writer key and the reader key for a fixture of inputs, including ids with special characters and empty components.
• •Treat any change to the constructor as a migration that requires a dual-read strategy and a documented grace period for the old key family.
• •Expose a key-level metric so drift causes a measurable regression instead of a silent miss, and alert on sudden growth in unprefixed or unexpected key families.
• •Version the key prefix visibly so a future change can be introduced without conflating old and new keys in the same keyspace.

Safe commands and checks

redis-cli -h <host> -p <port> --scan --pattern "<family-prefix>:*" | head
redis-cli -h <host> -p <port> GET "<writer-key-for-id>"
redis-cli -h <host> -p <port> GET "<reader-key-for-id>"
redis-cli -h <host> -p <port> INFO keyspace
redis-cli -h <host> -p <port> INFO stats | grep -E "keyspace_hit|keyspace_miss"
redis-cli -h <host> -p <port> TTL "<writer-key-for-id>"
redis-cli -h <host> -p <port> OBJECT ENCODING "<writer-key-for-id>"