Caching seems easy. Throw a Redis cluster in front of your database, set a TTL, and watch your p95 latency drop. But caching is a double-edged sword: it's the cheapest way to scale reads, and also the fastest way to serve wrong data to millions of users. I've seen caching bugs turn a 10-line fix into a 3-hour incident call.
This post covers the non-obvious failure modes of caching — not just 'remember to invalidate,' but the patterns that actually cause production incidents. I'll share three specific war stories and the patterns that could have prevented them.
War Story #1: The 15-Minute Grace Period That Lasted 3 Hours
At a previous company, we ran an e-commerce platform. We cached product availability in Redis with a 15-minute TTL. The idea was simple: if inventory changed, we'd wait up to 15 minutes for the cache to expire. That was fine for most products — but for flash sales, 15 minutes of overselling was catastrophic.
One Black Friday, a popular item went out of stock in the database after 1,000 units sold. But Redis still showed 'in stock' for the next 12 minutes. 3,400 customers placed orders that couldn't be fulfilled. The fix wasn't just shortening the TTL — we needed invalidation on write.
TTL-only invalidation is a ticking bomb for any data that changes asynchronously. If you can't tolerate 15 minutes of stale data, you need explicit invalidation.
# Before: TTL-only cache
redis.set(f"product:{product_id}:stock", stock, ex=900)
# After: invalidate on write
def update_stock(product_id, new_stock):
db.execute("UPDATE products SET stock = %s WHERE id = %s", (new_stock, product_id))
redis.delete(f"product:{product_id}:stock") # invalidate before TTL
# On read: cache-aside
def get_stock(product_id):
stock = redis.get(f"product:{product_id}:stock")
if stock is None:
stock = db.query("SELECT stock FROM products WHERE id = %s", product_id)
redis.setex(f"product:{product_id}:stock", 900, stock)
return stockWar Story #2: The Cross-Tenant Cache Leak
Another incident involved a multi-tenant SaaS platform. We cached user preferences in a single Redis instance using keys like 'prefs:{user_id}'. The problem? User IDs were not globally unique — tenant A's user 42 could overwrite tenant B's user 42. When a support agent changed preferences for one customer, another customer suddenly saw different UI.
We caught it because a customer complained that the dashboard was in Spanish — they had never changed the language. The fix: include the tenant ID in the cache key.
# Before: key collision across tenants
key = f"prefs:{user_id}"
# After: tenant-scoped key
key = f"prefs:{tenant_id}:{user_id}"The Pattern: Key Design Matters
Cache key collisions are a classic invalidation bug. If two different entities produce the same key, one will overwrite the other. This is especially common with integer IDs and auto-increment across tables. Always include a prefix that identifies the entity type and tenant.
War Story #3: The Read-Through Cache That Never Updated
We used a read-through cache with a library that would automatically populate the cache on miss. The database had a trigger that updated a 'last_modified' timestamp. Our cache used that timestamp in the key: 'profile:{user_id}:{last_modified}'. Every time the profile changed, the key changed, so old keys would never be accessed again. We thought we were clever.
But we forgot to handle deletes. When a user was deleted, the cache still had the old key — and since no one would ever request that user again, the stale entry sat there forever. Worse, if the user was re-created (e.g., GDPR deletion and sign-up again), the new profile had a different ID. The old key remained, causing ghost data in analytics.
Versioned keys are great for immutable objects, but you must also handle tombstone keys — use a TTL on all keys, even versioned ones, to automatically clean up orphaned entries.
Patterns to Prevent Cache Invalidation Bugs
- 1Write-Through Cache: Every write goes through the cache. The cache is updated atomically with the database. This ensures the cache is always consistent, but adds latency to writes.
- 2Cache-aside with Explicit Invalidation: Invalidate the cache entry before updating the database. This is the most common pattern and works well when reads far outnumber writes.
- 3TTL as a Safety Net: Always set a TTL, even if you invalidate explicitly. This prevents stale data from living forever if an invalidation is missed.
- 4Versioned Keys: Use a version number or timestamp in the key. On write, increment the version. Old keys are eventually evicted by LRU or TTL.
- 5Lease or Lock for Stampedes: Use a distributed lock (e.g., Redis Redlock) to prevent multiple processes from recomputing the same cache entry simultaneously.
Instrumentation is Your Early Warning System
You can't fix a bug you don't know about. Cache hit ratio is a coarse metric — it won't tell you if you're serving stale data. Instead, add a 'stale count' metric: every time you serve a cached value, also record the timestamp of when it was last written to the database. If the difference is above a threshold, increment a counter and log a warning.
Also, monitor cache invalidation calls. If the number of invalidations per minute drops suddenly, you might have a bug in your invalidation logic — or a client that stopped sending invalidation events.
of cache-related incidents in our postmortem database were caused by missing or incorrect invalidation.
Cache invalidation is not a single action — it's a system of alarms, keys, TTLs, and idempotency. One missing delete and you're serving lies.
Conclusion
Caching bugs are insidious because they don't crash your system — they silently corrupt the data your users see. The three war stories above cost us thousands in lost revenue and engineering hours. The patterns are simple but easy to forget: namespace your keys, invalidate on write, set TTLs, and instrument staleness.
Next time you add a cache, ask yourself: what happens if this cache entry is never invalidated? What if it's invalidated twice? What if two services try to invalidate at the same time? Answer those questions, and you'll avoid the worst caching bugs.
If you're using Redis, consider using Redis Streams or keyspace notifications to propagate invalidation events across services. This decouples cache invalidation from your write path and makes it auditable.
Frequently asked questions
What is a cache stampede and how do you prevent it?
A cache stampede happens when a cached key expires and many concurrent requests all miss the cache, then all try to recompute the value simultaneously, overwhelming the backend. Mitigations include: using a mutex/lock around the recomputation, adding jitter to TTLs, or using probabilistic early expiration (e.g., the 'XFetch' algorithm).
When should I use write-through instead of cache-aside?
Write-through cache is better when you need strong consistency between cache and database — every write goes through the cache first. Cache-aside is simpler and more performant for read-heavy workloads, but you risk stale data if you forget to invalidate after writes. I prefer write-through for inventory, balances, or any data where staleness means money lost.
How do I debug a cache invalidation bug in production?
Start by comparing the cached value and the database value for a known key. Add structured logging around cache get/set/invalidate operations with enough context (key, timestamp, caller). Use Redis MONITOR or memcached stats to trace operations. If you're using Redis, enable keyspace notifications to log all evictions and expirations.
What's the difference between cache invalidation and cache eviction?
Invalidation is an explicit action to remove or update a cached entry (e.g., after a write). Eviction is an automatic removal by the cache due to memory pressure (e.g., LRU, LFU). Invalidation bugs usually stem from missing explicit invalidation. Eviction bugs happen when hot keys get evicted prematurely, causing performance degradation.