02 / Failure modes
Recognize the system pattern behind the symptom.
Deep investigations into retries, locks, caches, queues, deadlines, and the failure loops that make incidents spread.
Cache stampedes: identify the expiry boundary that synchronized traffic
Cache stampede: many concurrent callers miss the same cache key at one expiry boundary and synchronously hit the origin, producing a step-shaped origin load spike that a steady miss rate cannot explain. This guide walks a backend engineer from the observable spike to a verified mitigation, without changing live cache state.
Open guide →Clock skew: diagnose timestamps that disagree across services
Clock skew debugging guide for distributed systems: how to detect disagreement among host clocks, isolate the synchronization layer responsible, and verify that an offset is within tolerance before changing any configuration.
Open guide →Connection-pool exhaustion: find the borrower that never returned
Connection-pool exhaustion happens when every reusable database connection stays checked out or blocked, so queued queries stall on `PoolExhausted` / timeout errors. This guide isolates the borrower that never returned its client, distinguishing leaks from saturation using pool metrics, then prescribes read-only triage before any code or config change.
Open guide →Duplicate background jobs: trace where idempotency was lost
Debug duplicate background jobs by tracing where idempotency was lost between the producer, the queue, the worker, and the side-effect target. Duplicates are common with at-least-once queues, retries, and concurrent workers, so the goal is to identify the layer that accepted or executed the same logical job more than once and add a deterministic guard before any irreversible work occurs.
Open guide →Lost updates: expose the read-modify-write race
Lost update is a read-modify-write race where a transaction reads a row, computes a new value from that stale read, and writes it back, silently overwriting a concurrent commit. This guide frames the failure boundary as the gap between the read snapshot and the write commit, and shows how to prove, isolate, and guard against it on PostgreSQL-style isolation levels.
Open guide →Partial writes: find the boundary that was not atomic
Partial-write debugging is the discipline of locating the exact boundary inside a multi-step operation where atomicity breaks and only a subset of the intended state is persisted. This guide walks backend engineers through a triage sequence that distinguishes logical partial commits from transport-level or transaction-level partial commits, and ties each suspect boundary to observable evidence rather than assumption. Output is a verifiable regression criterion, not a refactor recommendation.
Open guide →Queue starvation: distinguish no work from unreachable work
Queue starvation occurs when some jobs never receive worker capacity while other work proceeds normally. The core debugging task is to distinguish "no work exists" from "work exists but workers cannot reach it" — typically via partition assignment, per-partition lag, and rebalance history rather than aggregate throughput.
Open guide →Retry storms: find the feedback loop before it amplifies
Retry storms occur when client retry logic compounds an already-degraded dependency by adding load faster than it can recover. Each retried request occupies a worker thread, connection slot, and downstream call budget, so retries amplify rather than relieve pressure. The hallmark is a positive feedback loop: rising latency triggers more retries, which raises latency further until the dependency saturates. Mitigation requires breaking the loop with jittered, bounded backoff and load-shedding before the dependency collapses entirely.
Open guide →Stale cache entries: prove which write missed invalidation
Stale cache entries persist when a write to the source-of-truth did not trigger, or correctly trigger, a cache invalidate or overwrite. The guide frames the task as proving which write missed invalidation, not as a generic cache-tuning overview. It defines an evidence-first triage: capture the stale value, identify candidate writes, correlate write and invalidation events, and confirm the cache key contract.
Open guide →Stale distributed locks: diagnose ownership that outlives work
Diagnose and remediate distributed locks that remain held after the owning process crashed, was partitioned, or paused past its TTL. Covers ownership-evidence reasoning, fencing token verification, lease-vs-TTL gaps, and clock-drift pitfalls. Aligns with the canonical Redlock guidance and fencing-token literature.
Open guide →Timeout amplification: map the deadline budget across services
Timeout amplification occurs when nested service calls independently enforce their own deadlines and retries, causing the cumulative latency and retry budget consumed along a call chain to exceed the originating caller's deadline. This guide maps the deadline budget across services, identifies where budgets multiply, and provides a triage sequence for diagnosing latency cascades triggered by deadline propagation failures.
Open guide →