02 / Failure modes
Recognize the system pattern behind the symptom.
Deep investigations into retries, locks, caches, queues, deadlines, and the failure loops that make incidents spread.
Cache stampedes: identify the expiry boundary that synchronized traffic
Cache stampede: many concurrent callers miss the same cache key at one expiry boundary and synchronously hit the origin, producing a step-shaped origin load spike that a steady miss rate cannot explain. This guide walks a backend engineer from the observable spike to a verified mitigation, without changing live cache state.
Open guide →API 429 burst limit: map caller rate to the provider's response window
When an HTTP API returns 429, the failure is almost never "the API is down" — it is a burst-window rejection. This guide shows how to map the caller's actual request emission rate onto the provider's short-term rate policy so that the 429s stop being mysterious. The argument: a 429 only makes sense once you can read the provider's window, the requester's rhythm, and the gap between them.
Open guide →API 429 Retry-After ignored: find the client-side retry loop
Retrying before the provider's stated Retry-After window is a client-side loop, not a server-side outage. This guide frames 429 handling as a contract negotiation: the client must honor the recovery deadline the server publishes, and the loop must be located in the originating client, not blamed on the gateway.
Open guide →API 502 upstream reset: correlate gateway errors with connection teardown
Explains how to interpret an HTTP 502 Bad Gateway response when the upstream closes or resets the TCP connection before the gateway obtains a usable response, and how to correlate that error with observable connection teardown evidence at the gateway boundary.
Open guide →API circuit breaker open: separate dependency recovery from caller pressure
When a client HTTP circuit breaker is open, every outbound call short-circuits with a fast failure (often 503 from the client, or no request reaching the dependency). Engineers must distinguish dependency recovery (the upstream is actually back) from caller pressure (the breaker tripped because of caller-induced load or policy) before re-enabling traffic. This guide frames the failure mode, the evidence needed, and a safe re-close path.
Open guide →API connection pool leak: identify requests that do not release clients
Connections remain checked out from an HTTP client pool even after their request or error path completes, exhausting the pool, stalling new requests, and surfacing as timeouts or queue-length errors. This guide gives an intermediate engineer a defensible triage sequence: confirm the symptom in the pool, isolate the code path that leaks, and add release-in-finally coverage with an observable regression check.
Open guide →API timeout budget drift: locate the deadline mismatch across layers
API timeout budget drift happens when proxy, client, and upstream layers enforce different deadlines for the same request, so the layer that times out first is rarely the layer that caused the slowdown. This guide explains how to identify which boundary fires first, reconcile conflicting budgets, and prove the fix with observable timing evidence.
Open guide →Browser retained listeners: trace the cleanup path that keeps nodes alive
Browser retained listeners are the dominant cause of detached-DOM retention in modern single-page apps. This guide walks through identifying the cleanup path, instrumenting Performance and Memory APIs, and verifying that handlers and observers actually release their references when the owning UI disappears.
Open guide →Cache stampede lock bypass: identify concurrent refill paths
When protective locking around a cache miss fails, multiple workers refill the same key simultaneously rather than serializing on a single owner. This guide isolates which refill path bypassed the lock boundary, why the stampede still reaches the origin, and how to prove coordination is enforced again.
Open guide →Connection-pool queue starvation: find borrowers blocking unrelated work
Connection-pool queue starvation occurs when a subset of long-lived borrowers holds pool slots long enough that unrelated, shorter work cannot acquire connections. This playbook explains how to recognize the symptom from pool and database statistics, distinguish it from overload or slow queries, identify the blocking borrowers, and apply safe mitigations without abandoning the pool's concurrency model.
Open guide →CSRF token rotation drift: explain valid forms becoming invalid
CSRF token rotation drift happens when the browser and the application server disagree about which generated anti-forgery token is current, so a form that was just rendered is rejected on submit. The mismatch is rarely an attack signal and almost always a synchronization problem between cookie/session state and the token bound to the visible form.
Open guide →Docker healthcheck flapping: separate process health from probe timing
Docker healthcheck flapping occurs when a container's HEALTHCHECK directive causes the Docker daemon to toggle the service between healthy and unhealthy states. This guide separates two contracts: the in-process health signal emitted by the application, and the probe timing chosen in the Dockerfile or compose file. Engineers learn to read `docker inspect` health fields, match interval/timeout/retries/start_period to actual process readiness, and prove stability with a sustained observation window.
Open guide →GitHub Actions concurrency cancellation: find the run that terminated work
Concurrency cancellation in GitHub Actions is a deliberate workflow-control mechanism, not a bug: a `concurrency` group policy causes a newer or older run to cancel a run that is still executing. The diagnostic task is to identify which run was terminated, which group key triggered the cancellation, and which policy (`concurrency.cancel-in-progress`) was responsible, so engineers can decide whether the cancellation is expected (a re-run, a push to a long-lived branch, a fan-out) or unintentional (a shared key producing collisions across unrelated workflows). Evidence is found in the run's lifecycle payloads, the GraphQL `WorkflowRun` node, and the workflow YAML, not in the job logs of the cancelled run.
Open guide →JWT clock skew expiry: identify valid tokens rejected by time disagreement
A JWT that is cryptographically valid, correctly signed, and issued seconds ago is still rejected with a standard expiration-related error. The token's `exp` and `nbf` claims are evaluated against a clock that disagrees with the issuer's clock by enough to flip the time-based validity decision, so the verifier marks an otherwise good token as expired or not-yet-valid.
Open guide →Kubernetes startup probe race: diagnose checks that run before readiness
Kubernetes startup probes can race against application initialization when dependency readiness (config loading, cache warm-up, downstream handshakes) outlasts the probe contract. This guide explains how to distinguish a startup probe race from liveness flapping, narrow it with ordered evidence, and apply safe fixes that preserve probe semantics.
Open guide →Log sampling hides failure: preserve evidence without flooding output
Learn how log sampling in Clarity and OpenTelemetry pipelines can silently drop the rare error events you need for root-cause analysis, and how to configure tail-based or head-based strategies that preserve failure evidence without flooding output. This guide gives a beginner-friendly triage sequence, named configuration boundaries, and observable proof that error events survive the pipeline.
Open guide →Optimistic version lost update: prove which writer ignored the revision
A disciplined playbook for proving which writer ignored a revision in an optimistic version lost update. It sequences a 10-minute triage, shows how to read PostgreSQL's pg_stat_database statistics and pg_locks views, and defines the exact UPDATE row counts and serializable failures that name the writer.
Open guide →OAuth callback state loss: trace where correlation data disappears
OAuth callback state validation failures most often originate between the relying party's cookie layer and its backend session store, not inside the identity provider. This guide traces where the state parameter disappears across the redirect boundary and gives engineers a verification path that distinguishes cookie blocking, SameSite policy interaction, session-store expiration, and CSRF middleware mismatch.
Open guide →OpenTelemetry trace context loss: find the async boundary without propagation
Trace context loss in OpenTelemetry happens when a unit of work crosses an async or process boundary that the SDK does not automatically instrument, causing downstream spans to start a brand-new trace. This guide isolates the exact boundary where propagation broke, verifies that the upstream traceparent was actually emitted, and shows how to restore propagation without guessing.
Open guide →Transaction plus external side effect split-brain: locate the commit boundary
Diagnose split-brain incidents where a database commit and an external side effect (HTTP call, message publish, email send, payment capture) cannot be atomically rolled back together. Use commit-boundary inspection, PostgreSQL statistics views, and idempotency tokens to localize where state diverged.
Open guide →PostgreSQL autovacuum lag: connect dead tuples to query slowdown
PostgreSQL autovacuum lag is a failure mode where the autovacuum worker cannot keep up with dead-tuple generation, causing table statistics and storage to drift away from reality. The guide connects pg_stat_all_tables dead-tuple counters to query planner regressions, and explains how to confirm lag versus a workload spike before changing any GUC.
Open guide →PostgreSQL CTE plan regression: detect when materialization changed the path
CTE plan regression in PostgreSQL occurs when the planner's treatment of a common table expression boundary changes—most often flipping between inlining and materialization—so a query that previously reused a result now re-evaluates it (or vice versa). The failure is plan-shaped: row estimates, node order, and buffer counts diverge from a known-good baseline even when the SQL text is unchanged. The goal is to detect the boundary, compare plans across versions, and choose the least-disruptive remedy.
Open guide →PostgreSQL index not used: separate planner choice from missing index
PostgreSQL "index not used" is rarely a missing index. It is usually the planner choosing a sequential or alternate path because statistics, selectivity, or plan-cost inputs have drifted from the data. This guide separates planner choice from missing index by reading the same evidence the planner sees: pg_stats, EXPLAIN (ANALYZE) row estimates, and pg_stat_user_tables/indexes activity. It targets engineers diagnosing slow queries where an expected index is ignored, and provides ordered triage, conditional fixes, and proof-of-fix checks grounded in the PostgreSQL monitoring statistics documentation.
Open guide →PostgreSQL long idle transaction: find the session holding back cleanup
A long idle PostgreSQL transaction keeps a snapshot open and silently blocks autovacuum, tuple cleanup, and relation bloat. This guide shows how to read pg_stat_activity and pg_locks to identify the idle-in-transaction session, confirm what it is pinning, and safely terminate it without collateral damage.
Open guide →Parameter-sensitive query plan: explain fast and slow executions
PostgreSQL's optimizer can pick different plans for the same prepared statement when bind parameter values change the selectivity estimates. This article explains how to detect that the same query runs in milliseconds for one parameter and seconds for another, how to read EXPLAIN output to confirm plan divergence, and how to decide whether planning-time fixes, statistics work, or plan caching changes are appropriate.
Open guide →PostgreSQL replica replay lag: distinguish write pressure from apply blockage
Replica replay lag on a PostgreSQL streaming replica is the gap between bytes received via WAL streaming and bytes replayed by the startup process. The editorial argument: replay lag is not one failure but two, and the right response depends on which one you are in. Write pressure pushes the receiver's write queue; apply blockage stalls the startup process on a lock, a slow query, or a missing replica identity. Conflating them leads to misallocated mitigation.
Open guide →Queue partition hotspot: find uneven work distribution
Queue partition hotspot is a failure mode where a partitioning scheme (hash, key, tenant, region, or worker lane assignment) routes a disproportionate share of work to one worker, lane, or queue partition, starving siblings and increasing tail latency. This playbook walks through observable signals, a first-ten-minute triage, boundary inspection, and evidence-gated fixes that redistribute load rather than mask the imbalance.
Open guide →Queue poison message: isolate the payload that repeatedly fails
A triage playbook for isolating a single queue payload that repeatedly fails processing and exhausts retries or worker capacity without reaching a valid terminal state. The guide walks through observable signals, scoped evidence collection, and bounded decisions so the offending payload can be quarantined and the queue returned to a healthy state without speculative code changes.
Open guide →Queue visibility timeout expiry: explain duplicate delivery during slow work
Visibility timeout expiry is a Queues failure mode where a worker's processing time exceeds the configured invisibility window, causing the same job to be re-enqueued and delivered to another worker. The result is duplicate side effects, idempotency violations, and unstable throughput. BullMQ's stalled-jobs documentation describes this recovery path explicitly and provides the configuration surfaces engineers use to bound it.
Open guide →Double-submit race: identify concurrent requests that create one resource twice
Diagnose double-submit races where two equivalent create-resource requests pass a uniqueness check before either write is visible, producing duplicate rows. The playbook walks through observable symptoms, ordered triage in the first ten minutes, targeted evidence collection, conditional safe fixes, and observable proof-of-fix checks.
Open guide →React effect fetch race: stop older responses overwriting newer state
Playbook for diagnosing and fixing the classic React effect fetch race, where a slower response from an earlier input (effect, query, or dependency change) commits state after a newer response, leaving the UI showing data that does not match the current input. Covers identification, triage sequence, and evidence-conditional fixes grounded in React's documented effect semantics.
Open guide →Redis cache-key drift: find readers and writers using different contracts
When writers and readers of a Redis cache derive different keys for the same logical resource, reads silently miss or hit stale entries. The classic cache-aside pattern only works if every path into the cache agrees on the key contract; even a small serializer change, prefix change, or whitespace difference can split the namespace and make the cache look like a flaky datastore. This guide shows how to evidence cache-key drift from Redis itself, find every reader and writer, and decide whether to realign the code or to introduce a versioned key prefix.
Open guide →Redis hot key: diagnose one logical key dominating capacity
A Redis hot key is a single logical key that absorbs a disproportionate share of request volume, saturating one shard, one thread, or one CPU core while the rest of the dataset stays cold. The failure mode is observable as rising p99 latency, per-key throughput spikes, and cluster imbalance even when aggregate memory and ops/sec look healthy. This guide walks through identifying a dominating key from production or development evidence, separating it from generic CPU saturation, and applying conditional mitigations only after the key is confirmed.
Open guide →Redis invalidation race: show how an old value returns after a fresh write
Redis cache-aside invalidation races produce stale reads after fresh writes because the order of operations across the writer, the cache, and concurrent readers is not atomic. This guide frames the failure mode as a coordination defect, walks through observable symptoms, and gives an evidence-first triage sequence that distinguishes stale-fill races from delayed-invalidation races before any code change is made.
Open guide →Redis TTL refresh loop: find why stale values never expire
Diagnose and resolve a Redis TTL refresh loop where repeated reads or background refreshes continually extend a key's expiration, preventing intended cache invalidation and leaving stale values alive far longer than the configured TTL.
Open guide →Retrying a non-idempotent side effect: find the duplicate operation
Timeouts on operations with real-world side effects (charges, transfers, sends) produce duplicate work when clients retry without knowing whether the first attempt completed. This guide shows how to detect that a duplicate actually executed, separate duplicate execution from duplicate delivery, and apply idempotency keys, request IDs, and conditional reconciliation before another retry is triggered.
Open guide →Retries without jitter: map synchronized clients to the load spike
When clients retry failed requests on identical backoff schedules, they re-synchronize and generate load spikes that can repeatedly take a dependency down. This guide shows how to recognize thundering-herd retry patterns, separate them from genuine capacity exhaustion, and verify that any mitigation actually desynchronizes clients. The argument: in distributed systems, retry logic without jitter converts transient faults into correlated cascades, so the proof of any fix must be measured in the variance of request arrivals, not just in a drop in error counts.
Open guide →Session cookie SameSite mismatch: diagnose cross-site login behavior
SameSite cookie attribute on session cookies can silently suppress delivery on cross-site navigations and embedded contexts, breaking OAuth/OIDC callbacks and authenticated API calls. This guide explains the delivery rules, how to recognize the failure in evidence, and how to verify the chosen SameSite policy actually carries the session across the boundary your flow uses.
Open guide →Turborepo environment input omission: explain incorrect cache reuse
Turborepo caches task outputs by hashing inputs declared in the task's pipeline definition, including environment variables explicitly listed via `env`. When a task's output actually depends on a variable that is not included in that list, the cached artifact is keyed without that variable's value, and a later run with a changed variable will reuse stale output as if nothing changed. This is the "environment input omission" failure mode: cache hit rate stays high while semantic correctness collapses.
Open guide →Vite cache invalidated too late: trace stale module output
This guide explains how to diagnose Vite's dev server continuing to serve transformed module output after an input file or configuration change has occurred, focusing on tracing stale module output through Vite's caching layers and dependency graph.
Open guide →Worker acknowledges before commit: expose the loss window
A playbook for diagnosing the "ack-before-commit" failure pattern in queue workers, where a job is acknowledged, removed, or marked complete before its database writes or external side effects are durable. The loss window between ack and commit is exposed through stalled-job recovery, duplicate processing, and reconciliation drift. Engineers use BullMQ stalled-job telemetry and worker lock semantics as the authoritative anchor for this guide.
Open guide →Clock skew: diagnose timestamps that disagree across services
Clock skew debugging guide for distributed systems: how to detect disagreement among host clocks, isolate the synchronization layer responsible, and verify that an offset is within tolerance before changing any configuration.
Open guide →Connection-pool exhaustion: find the borrower that never returned
Connection-pool exhaustion happens when every reusable database connection stays checked out or blocked, so queued queries stall on `PoolExhausted` / timeout errors. This guide isolates the borrower that never returned its client, distinguishing leaks from saturation using pool metrics, then prescribes read-only triage before any code or config change.
Open guide →Duplicate background jobs: trace where idempotency was lost
Debug duplicate background jobs by tracing where idempotency was lost between the producer, the queue, the worker, and the side-effect target. Duplicates are common with at-least-once queues, retries, and concurrent workers, so the goal is to identify the layer that accepted or executed the same logical job more than once and add a deterministic guard before any irreversible work occurs.
Open guide →Lost updates: expose the read-modify-write race
Lost update is a read-modify-write race where a transaction reads a row, computes a new value from that stale read, and writes it back, silently overwriting a concurrent commit. This guide frames the failure boundary as the gap between the read snapshot and the write commit, and shows how to prove, isolate, and guard against it on PostgreSQL-style isolation levels.
Open guide →Partial writes: find the boundary that was not atomic
Partial-write debugging is the discipline of locating the exact boundary inside a multi-step operation where atomicity breaks and only a subset of the intended state is persisted. This guide walks backend engineers through a triage sequence that distinguishes logical partial commits from transport-level or transaction-level partial commits, and ties each suspect boundary to observable evidence rather than assumption. Output is a verifiable regression criterion, not a refactor recommendation.
Open guide →Queue starvation: distinguish no work from unreachable work
Queue starvation occurs when some jobs never receive worker capacity while other work proceeds normally. The core debugging task is to distinguish "no work exists" from "work exists but workers cannot reach it" — typically via partition assignment, per-partition lag, and rebalance history rather than aggregate throughput.
Open guide →Retry storms: find the feedback loop before it amplifies
Retry storms occur when client retry logic compounds an already-degraded dependency by adding load faster than it can recover. Each retried request occupies a worker thread, connection slot, and downstream call budget, so retries amplify rather than relieve pressure. The hallmark is a positive feedback loop: rising latency triggers more retries, which raises latency further until the dependency saturates. Mitigation requires breaking the loop with jittered, bounded backoff and load-shedding before the dependency collapses entirely.
Open guide →Stale cache entries: prove which write missed invalidation
Stale cache entries persist when a write to the source-of-truth did not trigger, or correctly trigger, a cache invalidate or overwrite. The guide frames the task as proving which write missed invalidation, not as a generic cache-tuning overview. It defines an evidence-first triage: capture the stale value, identify candidate writes, correlate write and invalidation events, and confirm the cache key contract.
Open guide →Stale distributed locks: diagnose ownership that outlives work
Diagnose and remediate distributed locks that remain held after the owning process crashed, was partitioned, or paused past its TTL. Covers ownership-evidence reasoning, fencing token verification, lease-vs-TTL gaps, and clock-drift pitfalls. Aligns with the canonical Redlock guidance and fencing-token literature.
Open guide →Timeout amplification: map the deadline budget across services
Timeout amplification occurs when nested service calls independently enforce their own deadlines and retries, causing the cumulative latency and retry budget consumed along a call chain to exceed the originating caller's deadline. This guide maps the deadline budget across services, identifies where budgets multiply, and provides a triage sequence for diagnosing latency cascades triggered by deadline propagation failures.
Open guide →