Retry storms: find the feedback loop before it amplifies

The symptoms

• •Error rate on a downstream dependency climbs non-linearly while its request rate also rises — both curves trend up together, indicating the client is generating load against an already-sick service.
• •Client-side latency p99 increases in lockstep with retry attempts per request, often visible as a sawtooth pattern where each timeout boundary produces a fresh burst.
• •Thread pool, connection pool, or async task queue utilization on the calling service climbs toward saturation even though user-facing inbound traffic is flat or falling.
• •Queue depth, listen-socket backlog, or kernel accept queue drops grow on the dependency despite no change in legitimate user load.
• •Logs show repeated identical requests with near-identical timestamps from the same caller, often with the same correlation ID retried 3–10 times within seconds.
• •Health-check endpoints on the dependency return 200 yet real request handling is degraded, because the dependency is overwhelmed by the very probes and retries hitting it.

Likely causes

• •Retry policy without jitter: clients retry at fixed intervals, so a synchronized retry wave hits the dependency the instant it begins recovering, re-overloading it.
• •Unbounded retry count or unbounded per-request timeout summed across attempts, so a slow dependency causes each caller to hold resources through many attempts.
• •No circuit breaker: failures are retried even after the dependency has crossed a known failure threshold, so the client keeps adding load during an outage.
• •Retry-on-5xx semantics that include 500/502/503/504 for retryable classes but do not exclude the case where the dependency is returning 503 because it is overloaded.
• •Idempotency gaps: clients retry non-idempotent operations without an idempotency key, so the retry storm also causes duplicate side effects, not just extra load.
• •Health checks, metrics scrapes, or readiness probes are themselves retried on transient error, multiplying internal load against the same dependency.
• •Shared retry middleware applied uniformly to user traffic and to internal control-plane calls, so an internal probe storm looks like a user-facing retry storm.

First ten minutes

1. 01Confirm scope: pick one dependency suspected of being overwhelmed and one caller suspected of amplifying load; do not start by examining multiple services at once.
2. 02Capture current inbound QPS, error rate, and p50/p99 latency on the caller for the suspected dependency, and the same three metrics on the dependency itself, before changing anything.
3. 03Compare the dependency's request rate against its pre-incident baseline; if the rate is elevated while user traffic is flat, the extra load is from retries, health checks, or another internal caller.
4. 04Sample 5–10 failed requests on the caller and record: HTTP status received, total attempts logged for that correlation ID, and the time gap between attempts, to characterize the retry pattern.
5. 05Read the retry configuration in the caller's client library or service mesh policy and note the maximum retry count, base delay, and whether jitter is enabled.
6. 06Stabilize first: if a circuit breaker exists, open it for the affected dependency to stop new retries immediately, rather than tuning retry numbers under live load.
7. 07Notify: record the dependency, caller, and evidence collected so far in the incident channel before deeper diagnosis, so a second responder does not restart the same investigation.

Evidence to collect

• •Caller-side metric: requests per second to the dependency, broken down by status code class (2xx, 4xx, 5xx, timeout), with attempt count per correlation ID.
• •Dependency-side metric: inbound requests per second, p50/p99 latency, thread or worker pool utilization, queue depth, and CPU saturation indicators.
• •Retry configuration snapshot: max attempts, base delay, max delay, jitter setting (full vs equal vs none), and whether per-attempt timeout exists independently of total deadline.
• •Log samples showing the same correlation ID retried multiple times with timestamps and the status that triggered each retry.
• •Time-aligned overlay of caller retry spikes against dependency latency spikes, to confirm the feedback relationship rather than coincidental correlation.
• •List of all internal callers of the dependency (other services, schedulers, health-checkers) so each source of retry traffic can be attributed.

Where to look

• •The client-side retry middleware or service mesh retry policy for the dependency: configuration files, environment variables, or sidecar annotations defining maxAttempts, retryOn, perTryTimeout, and retryBackoff.
• •The caller-side outbound HTTP or RPC client wrapper: timeout settings, connection pool size, and any decorator that re-issues requests on exception.
• •The dependency's request admission boundary: ingress controller metrics, gateway logs, application server accept queue, and thread pool stats.
• •Shared message buses or queues in front of the dependency: queue depth, consumer lag, and redelivery count, which may themselves be a retry mechanism masquerading as load.
• •The dependency's own dependency graph one hop downstream: a retry storm at level N often originates from level N+1 failing and re-driving N with retries.
• •Observability platform queries correlating caller retry counters with dependency latency histograms over the same time window.

Diagnostic steps

1. 01Plot caller outbound QPS to the dependency against dependency inbound QPS; if they rise together while upstream user traffic is flat, retries are a primary contributor.
2. 02Histogram retry attempts per request on the caller; a distribution concentrated at the configured maxAttempts (e.g. 5 or 10) indicates retries are being exhausted, not abandoned early.
3. 03Measure the inter-attempt delay distribution; a tight cluster near a single value implies no jitter and predicts synchronized retry waves on partial recovery.
4. 04Inspect dependency 5xx and 503 rates: if 503 is being returned specifically for overload, check whether the retry policy treats 503 as retryable; if so, the policy is working against recovery.
5. 05Check whether a circuit breaker is present and its state over the incident window; an absent or closed breaker during rising error rate is the structural cause of amplification.
6. 06Compare the dependency's saturation indicator (thread pool, event loop lag, queue depth) against its request rate; saturation rising faster than rate indicates requests are slow, not just numerous.
7. 07Trace a single failing user request end-to-end and count downstream calls; if one logical request expands to many physical calls, fan-out plus retries is multiplying load multiplicatively.
8. 08Decide between two hypotheses: (a) caller retries are the dominant amplifier, or (b) another internal system is the dominant amplifier; do not change retry config until this is settled.

Common mistakes

• •Increasing retry counts or extending timeouts during an active storm, on the assumption that more retries will get through — this deepens the amplification and worsens recovery time.
• •Adding retries to a path that has none, as a "make it more reliable" change, without modeling the multiplicative effect against the dependency's capacity.
• •Treating health-check failures as transient and retrying them aggressively; this converts a probe into a load generator against the same dependency being diagnosed.
• •Removing jitter "to make timing deterministic for debugging," which then produces synchronized waves that confirm jitter's value by causing a visible collapse.
• •Disabling retries globally rather than only for the degraded dependency, which masks the failure mode and leaves the dependency unprotected on the next incident.
• •Assuming the dependency "is unhealthy because the caller is slow," inverting cause and effect; the dependency is slow because the caller (and others) are sending too much traffic.

Safe fixes

• •If a circuit breaker exists for the dependency, open it manually for the affected route so new retries stop immediately; verify by watching caller outbound QPS to the dependency fall within one interval.
• •Enable or strengthen jitter on the retry policy: full jitter is preferable to equal jitter when the dependency is near saturation; verify by checking the inter-attempt delay distribution broadens.
• •Cap the total deadline across attempts, not just the per-attempt timeout, so a single caller cannot hold resources through unbounded retries; verify by inspecting p99 caller latency against the new budget.
• •Reduce max retry attempts to a small number (commonly 2–3) for the degraded dependency specifically, scoped via route or client config rather than a global default; verify via retry-attempt histogram.
• •Exclude 503 from the retryable set when 503 is being used as an explicit overload signal by the dependency, or honor a Retry-After header instead of a fixed backoff; verify by sampling 503 responses.
• •Apply bulkhead isolation so retries against one dependency cannot exhaust the caller's connection pool for other dependencies; verify by monitoring pool utilization per dependency.
• •Dampen internal retries first (health checks, metrics scrapes, schedulers) before tuning user-facing retries, since internal loops often dominate load against a sick dependency.

Prove the fix

1. 01Dependency inbound QPS returns to within an agreed band of its pre-incident baseline while user-facing traffic remains at current levels; sustained for at least one full retry-cycle window.
2. 02Retry-attempt histogram on the caller shifts left: the mass moves from maxAttempts toward 1 attempt, and the inter-attempt delay distribution shows spread consistent with enabled jitter.
3. 03Dependency p99 latency trends downward over consecutive measurement intervals after circuit-breaker open and retry tuning, with no synchronized retry spike visible at any single timestamp.
4. 04Circuit breaker transitions to half-open and closes without re-opening on a synthetic fault injection that elevates dependency latency briefly; this confirms the breaker, not luck, broke the loop.
5. 05A chaos test that introduces dependency slowness and verifies retries do not exceed a defined amplification factor (e.g. caller QPS to dependency ≤ N× baseline) passes reproducibly.
6. 06Synthetic load replay: replay the same inbound traffic shape against the new retry policy and confirm dependency saturation indicators remain below the threshold that previously triggered the storm.

Prevention and next steps

• •Adopt jittered, bounded retries as a default contract for every outbound client; require a documented maxAttempts, base delay, max delay, jitter mode, and total deadline per dependency.
• •Require a circuit breaker on every external or cross-service dependency, with explicit open/half-open thresholds tied to the dependency's SLO rather than fixed counts.
• •Separate retry configuration per dependency class (critical user path, internal control plane, batch) so a storm in one class cannot borrow another's retry budget.
• •Add dashboards that overlay caller outbound QPS, dependency inbound QPS, and dependency saturation on a single chart, so amplification is visible before collapse.
• •Review idempotency for any operation that may be retried; without an idempotency key, a "successful" retry storm can corrupt state even after the load problem is fixed.
• •Include retry and circuit-breaker behavior in load tests and game days; a system that has never failed under retry pressure cannot be assumed safe in production.

Safe commands and checks

Read caller retry config from the client wrapper: grep -nE "maxAttempts|retryOn|perTryTimeout|retryBackoff|jitter" <client-config-file>
Count retries per correlation ID in caller logs: awk '/<correlation-id-field>/{c[$field]++} END{for(k in c) if(c[k]>1) print k,c[k]}' <caller-log-file> | sort -k2 -nr | head -20
Compute inter-attempt delay spread for retried requests: awk 'BEGIN{prev=0} /<correlation-id-field>/{if(prev){print $timestamp-prev}; prev=$timestamp}' <caller-log-file> | sort -n | uniq -c | head -20
Inspect dependency saturation indicators: ss -s (socket summary) and for the process pid <pid> read /proc/<pid>/status and /proc/<pid>/io to observe threads, context switches, and blocked state.
Histogram status codes returned by the dependency to the caller: awk '/<status-field>/{c[$status]++} END{for(k in c) print k,c[k]}' <caller-log-file> | sort -k2 -nr
List all internal callers of the dependency via service registry or config: grep -rn "<dependency-name>" <service-config-root> | grep -E "host|url|endpoint"