All playbooks

Playbook 08 / 17

Debugging Tenant Cache Bugs

Tenant cache bugs are both correctness and security incidents.

The pattern

A cache key is built without the tenant or user scope, so the first tenant to warm the cache decides what every other tenant sees. It looks like a correctness bug but it is also a data-isolation incident. These bugs hide because they only appear after a warm cache, and the wrong data is perfectly valid - for someone else.

( 01 )Symptoms

How this failure announces itself.

  • warningA user reports seeing another account's data, intermittently.
  • warningThe bug disappears on cache flush and slowly returns as traffic warms keys back up.
  • warningA security report traces back to a performance optimization that added caching.
( 02 )First moves

The first ten minutes — establish facts before touching code.

  • 1Take one cross-tenant report and find the exact cache key that served the wrong value.
  • 2List the dimensions in that key versus the dimensions the value depends on - the missing one is the bug.
  • 3Audit sibling keys built in the same module for the same omission; this bug travels in groups.
  • 4Treat it as an incident: estimate how long the key was shared and which tenants were exposed.
( 03 )Where to look

The code and config that usually owns this bug.

  • searchThe cache key builder - list every dimension the value depends on and compare against what is actually in the key.
  • searchTenant context propagation - does the tenant id reach the layer building the key, or is it lost inside a helper?
  • searchShared process memory - module-level memoization is a cache too, and it ignores tenancy by default.
  • searchInvalidation paths - per-tenant writes must not leave other tenants' stale entries behind.
( 04 )Common fixes

Fix the cause, then make the regression impossible.

  • buildInclude the tenant id in every key for tenant-scoped values - no exceptions, even for global-looking data.
  • buildCentralize cache key construction in one helper that requires a scope argument, so omission becomes a type error.
  • buildAdd cross-tenant regression tests: warm the cache as tenant A, read as tenant B, assert isolation.
  • buildDocument which caches are intentionally global and why, so review can challenge the list.
( 05 )Prove the fix

A fix you can't demonstrate is a guess. Close the loop.

  • verifiedThe warm-as-A-read-as-B test fails on the old code and passes on the fix.
  • verifiedInspect live keys (or key logs) to confirm tenant ids appear where expected.
  • verifiedConfirm per-tenant invalidation evicts only that tenant's entries.