Retries without jitter: map synchronized clients to the load spike

The symptoms

• •Error rates spike in sharp, periodic bursts every N seconds instead of decaying smoothly after an incident, with the period matching the configured (or default) client retry interval.
• •Upstream or origin service returns 502 Bad Gateway in rhythmic waves even though peak request volume does not exceed documented capacity, and the waves correlate with the previous error burst by exactly one retry interval.
• •Load balancer or connection-level metrics (requests per second, new connections, queue depth) show a sawtooth or comb pattern rather than a smooth curve, with each tooth separated by the retry timeout used by the client library.
• •Multiple independent client populations (different services, regions, or SDK versions) all begin failing at the same wall-clock offset after a single upstream hiccup, indicating shared timing rather than independent cause.
• •Tail latency and error budgets degrade only during the spike windows; aggregate throughput may look healthy on a 5-minute average, masking the synchronized pattern visible at 1-second or 100-millisecond resolution.

Likely causes

• •Retry policy uses a fixed delay (for example, a constant 1s or 2s sleep) with no randomization component, so every client instance waits the same interval and re-fires simultaneously.
• •Multiple clients or SDKs independently choose a default retry interval (commonly 1 second) because the platform or framework did not expose jitter as a configurable option, producing a herd with no shared coordinator.
• •A circuit breaker or rate limiter has a fixed cooldown period that all callers honor identically, so when it half-opens, the full client population probes at the same instant.
• •Per-host retry budgets are large relative to peer count, so even a sub-second synchronization between clients is enough to push an already-warm dependency past its connection or thread ceiling on the next wave.
• •Server-side health checks and client retries share a clock or heartbeat cadence, so a missed health check reissues work synchronously across all attached clients at the next tick.

First ten minutes

1. 01Confirm the failure shape is periodic, not organic: take the last hour of request-rate and error-rate series and bucketize at 1-second resolution; aligned spikes separated by an identical interval are the strongest single signal of synchronized retries.
2. 02Identify the canonical client retry interval in use by inspecting the SDK, framework, or proxy configuration (for example, HTTP client retry-after, gRPC default backoff, service mesh outlier detection cooldown) and record both the configured delay and whether a jitter parameter exists.
3. 03Correlate the observed spike period against that retry interval; a match within tens of milliseconds is strong evidence that the bursts are retry-driven rather than capacity-driven.
4. 04Snapshot current connection-pool saturation, thread-pool occupancy, and queue depth on the affected upstream at a spike peak so you can distinguish "too many simultaneous retries" from "steady-state overload" before changing anything.
5. 05Pull a small sample of failed requests and verify the upstream HTTP status is 502 Bad Gateway (as documented for that response code), with response headers and timing consistent across the burst rather than scattered across many error types.
6. 06Decide on a working hypothesis before any client change: synchronized retries vs. genuine capacity ceiling vs. shared health-check cadence; the remaining steps depend on which hypothesis survives this triage.

Evidence to collect

• •Time series of request rate, error rate, and a specific error code (502 Bad Gateway count) at the finest resolution your metrics system supports, covering at least the last full burst cycle.
• •The configured retry policy on each distinct client population, including base delay, maximum delay, maximum attempts, and whether any jitter (full, equal, decorrelated) is enabled.
• •Upstream resource counters at peak: open connections, in-flight requests, thread or worker pool utilization, and queue depths, sampled at the same timestamps as the spike.
• •A sample of failing requests showing the upstream response status, the time elapsed since the original request, and the client instance identifier, to confirm retries rather than independent new requests.
• •Distribution of inter-arrival times between requests to the failing upstream during steady state versus during the spike, to quantify how synchronized the load actually is.

Where to look

• •Client boundary: the HTTP/gRPC client library configuration in each service that calls the failing upstream, including any wrapper or resilience library that sets default retry values; a single shared library version across many services is a common synchronization source.
• •Service mesh or API gateway boundary: outlier detection, retry policy, and per-host retry budgets applied uniformly across routes; mesh-level retries often lack jitter by default.
• •Load balancer and reverse proxy boundary: retry behavior, connection draining, and any request queuing layer that re-injects work on a fixed cadence.
• •Upstream service boundary: connection accept limits, worker pool size, per-IP connection caps, and any backpressure or shed-load signal that exposes the synchronized requests as 502 responses.
• •Health-check boundary: the cadence and source of liveness or readiness probes from orchestrators (Kubernetes, load balancers) and any application-level health endpoints, since synchronized probes and synchronized retries share the same failure shape.

Diagnostic steps

1. 01Bucketize request arrivals at 1-second (or 100-ms, if available) resolution and compute the autocorrelation of the error series; a strong peak at lag equal to the client retry interval distinguishes synchronized retries from a smooth overload curve.
2. 02Compare the period of the observed burst against three candidate clocks in this order: client retry delay, circuit breaker or mesh outlier cooldown, and upstream health-check interval; the candidate whose period matches the burst within tolerance is the operative one.
3. 03From a sample of failed requests, compute the distribution of "time since original request" seen by the upstream; in true retry storms this is clustered near the configured delay, whereas in independent errors it is spread across the full latency range.
4. 04Inspect the upstream's resource counters at the spike peak: if in-flight requests or open connections exceed the documented ceiling exactly during the spike and recover between spikes, the constraint is being hit transiently by the herd, not exhausted in steady state.
5. 05Rule out genuine capacity ceiling by checking whether peak QPS (averaged over a full spike window, not a single second) stays within the upstream's documented limits; if average demand is under capacity but instantaneous demand is not, the failure mode is synchronization, not throughput.
6. 06Rule out health-check synchronization by disabling or extending the probe interval in a non-production environment and observing whether the burst period shifts to match the new interval; do not apply this as a production change without a controlled test.
7. 07Confirm by controlled re-test: if you can shift one client population's retry delay by a known offset and observe its burst phase shift accordingly, the herd is retry-driven and not a capacity artifact.

Common mistakes

• •Concluding that adding more capacity will fix the failure when the average QPS is already within limits; a synchronized herd can still overrun an oversized pool because the arrivals concentrate in a sub-second window.
• •Enabling jitter on only some clients while others keep fixed delays; partial desynchronization can still produce a smaller but still periodic spike and mislead the team into thinking the fix is insufficient when the real issue is incomplete rollout.
• •Increasing retry counts without bounding total request volume; without a retry budget or global cap, jitter alone converts a sharp spike into a longer, lower herd that may still saturate the upstream.
• •Adjusting server-side timeouts or pool sizes to "absorb" the herd without addressing the source; tuning the downstream can mask the symptom temporarily and delay the diagnosis of the client-side synchronization.
• •Trusting 5-minute aggregated dashboards that average the bursts into a smooth curve; the synchronization is invisible at coarse resolution and only appears at second or sub-second granularity.

Safe fixes

• •Add full or equal jitter to the client retry delay so each instance's wait is uniformly distributed within a defined window, and document both the base delay and the jitter window; this is the minimum viable desynchronization.
• •Cap total retries with a per-request retry budget (for example, a token bucket per upstream per client) so even if jitter spreads the load, the aggregate retry volume cannot exceed a known fraction of new traffic.
• •Stagger any circuit-breaker half-open probe across clients using jittered cooldowns, so the moment a dependency recovers is not also the moment every client hammers it; verify the half-open cadence in configuration before rollout.
• •If a service mesh or shared library enforces retries uniformly, enable jitter at the mesh layer and verify the new policy is actually applied to the routes serving the failing upstream, since policy drift is a common silent failure here.
• •For health-check-induced synchronization, decouple probe cadence from request retry cadence and add jitter to probes only after confirming the source, because changing probes affects failure detection latency as well.
• •Roll the change out to one client population first, behind a flag if available, so you can compare the desynchronized series against a control before extending the change.
• •Each fix is conditional on a specific piece of evidence: jitter only after the burst period has been matched to a retry interval; retry budget only after aggregate retry volume is shown to be the operative constraint; mesh-level jitter only after confirming the mesh policy is the active retry source.

Prove the fix

1. 01After enabling jitter, the autocorrelation of the error series at the previous "retry interval" lag should drop by orders of magnitude relative to the pre-change baseline; a small reduction is not a fix.
2. 02The inter-arrival time distribution of requests to the failing upstream during what was previously a spike window should become measurably more uniform, with the standard deviation of the per-second count increasing while the peak count decreases.
3. 03The peak instantaneous in-flight request count during a spike window should fall below the upstream's documented connection or thread ceiling, even if the average QPS is unchanged.
4. 04Introduce a controlled fault (for example, a brief upstream brownout in a staging environment) and observe whether the recovery curve is smooth rather than sawtoothed; this is the direct regression test for the failure mode.
5. 05Verify that the 502 Bad Gateway rate during the post-fault window matches the rate of independent, non-retry-induced errors expected from the fault itself, with no periodic component visible in a 1-second-resolution plot.
6. 06Run the proof against at least one full burst cycle, and capture both the pre-change and post-change series at the same resolution so the comparison is apples-to-apples; document the metrics, the window, and the client populations involved.

Prevention and next steps

• •Standardize a default client retry policy across services that mandates jitter, a bounded maximum delay, and a retry budget, and make the configuration a reviewable artifact rather than implicit defaults.
• •Add a pre-deployment check that flags any retry configuration lacking a jitter parameter or exceeding a per-host retry budget, integrated into service configuration review or CI.
• •Instrument second-resolution request and error metrics for every dependency your services call, so synchronized retry patterns are visible before they cascade into user-visible 502 responses.
• •Document the upstream connection, thread, and queue ceilings alongside the SLO so capacity and synchronization are reasoned about on the same page, and so the distinction between "too much load" and "too synchronized load" stays explicit.
• •Periodically rehearse a controlled upstream brownout in a non-production environment and use the resulting error series as a regression signal for retry-induced synchronization.

Safe commands and checks

awk '{print strftime("%H:%M:%S",$1), $2}' access.log | sort | uniq -c | awk '{print $1}' | uniq -c | sort -rn | head — count requests per second from an access log; a single high count followed by repeats at a fixed interval is the script-level analog of the autocorrelation check.
grep -c ' 502 ' access.log — count 502 Bad Gateway responses in the current log window as a quick sanity check that the working file contains the response code cited in the official HTTP status reference.
grep ' 502 ' access.log | head — inspect a small sample of 502 responses to confirm upstream status, client identifier, and timing match the burst rather than scattered independent errors.
grep -E '"(retry|backoff|jitter)"' config.yaml | sort -u — surface the actual retry-related keys present in a service configuration file; absence of any "jitter" key is the simplest textual signal of the failure mode.
awk '/^retry_delay/ {print}' config.yaml — extract the configured retry delay value(s) verbatim so the observed burst period can be cross-checked against them in triage.