API 429 Retry-After ignored: find the client-side retry loop

The symptoms

• •Spike in outbound request rate from one service or worker even though the upstream provider returned 429 with a Retry-After header on the previous response
• •Trace data shows a second request to the same upstream endpoint arriving seconds (or sub-seconds) after a 429 response, inside the Retry-After window the server advertised
• •Upstream provider reports throttling, quota exhaustion, or temporary blocks while the local service logs show no parse of the Retry-After header value
• •Backoff is visibly uniform or jitter-free, suggesting a fixed delay rather than a header-driven delay
• •User-visible symptoms include intermittent 5xx leakage to clients, elevated p99 latency, or queue saturation in the calling service rather than steady failures

Likely causes

• •An HTTP client default or middleware layer silently drops the Retry-After response header before the retry policy can read it
• •Retry policy is configured for a different status code (for example 503) and never matches the 429 branch, so the policy falls through to a default short delay
• •Multiple retry layers coexist: a transport-level retry, an SDK retry, and an application-level retry, each with its own delay logic that ignores the header
• •Retry-After is parsed as a number of seconds but the server sent an HTTP-date form, or vice versa, and the parser returns 0 or NaN
• •Polling workers, cron jobs, or queue consumers re-enqueue the request on a fixed cadence that bypasses the HTTP retry path entirely
• •A shared circuit breaker or rate limiter upstream of the client resets its budget faster than the provider's recovery window, allowing retries to flow before the deadline

First ten minutes

1. 01Confirm the symptom is a client-side loop, not a server-side outage: capture one 429 response and verify the Retry-After header is present and non-empty in the raw response
2. 02Identify the calling service boundary and the single client component that originated the retried request using trace IDs that span the retry attempt
3. 03Pull the retry policy configuration for that client and check whether 429 is in the matched status code set and whether the delay is bound to the header
4. 04Search the codebase for any second retry layer (SDK, transport, application) that could issue a request without consulting the header value
5. 05Compare the advertised Retry-After value against the observed inter-request gap on the wire to prove the gap is smaller than the header dictates

Evidence to collect

• •Raw HTTP responses for the failing endpoint showing the Retry-After header, its format (delta-seconds or HTTP-date), and the response date
• •Trace or log records that pair the 429 response with the next outbound request to the same endpoint, including timestamps accurate to the second
• •The retry policy configuration object or code path that handles 429, including the exact delay expression used when the header is present
• •List of all retry-capable layers in the call stack: transport, SDK, middleware, application-level decorator, job scheduler
• •Provider documentation or developer console values that confirm the Retry-After semantics, including whether the value is per-request or per-account

Where to look

• •At the HTTP client boundary: the request builder and response interceptor where Retry-After would normally be read
• •At the SDK retry decorator: the policy object that maps status codes to action and delay
• •At the middleware chain: any layer that strips, rewrites, or consumes response headers before the retry policy runs
• •At the job scheduler or queue consumer: the cadence at which work is re-enqueued, which can bypass the HTTP retry path
• •At the provider developer console: rate limit dashboards and any documentation page stating how Retry-After values are computed

Diagnostic steps

1. 01Reproduce the loop deterministically: issue a known rate-limited request and record the elapsed wall-clock time between the 429 response and the next outbound request to the same endpoint
2. 02Diff the elapsed gap against the Retry-After value from the response; if the gap is strictly less than the header, the client is not honoring the window
3. 03Inspect the retry policy object and confirm whether the delay resolver reads the header or substitutes a constant; a constant is sufficient evidence the header is ignored
4. 04Walk the request through each layer in the call stack and record, for each layer, which headers it inspects and which it strips; the first layer that loses the header is the boundary to fix
5. 05Check parser handling for both delta-seconds and HTTP-date forms; induce one of each in a controlled environment and confirm the parser returns a non-zero positive duration
6. 06Search for any global rate limiter, concurrency limiter, or circuit breaker that schedules a retry independently of the HTTP response; such a layer can override the header-driven delay
7. 07Correlate the loop with worker or job schedules to rule out a polling cadence that reissues the request outside the HTTP retry path

Common mistakes

• •Concluding the provider is misbehaving when the client never read the Retry-After header at all; the absence of honoring is local, not remote
• •Assuming a single retry layer exists; layered clients often have a transport retry and an SDK retry, and either one can ignore the header
• •Treating Retry-After as advisory and applying a fixed exponential backoff; the header is the contractually declared recovery window, not a suggestion
• •Increasing concurrency or worker count to mask the symptom, which typically amplifies the throttling and shortens the recovery window further
• •Parsing Retry-After as a delta when the server sent an HTTP-date, producing a near-zero delay and a tight loop

Safe fixes

• •Bind the retry delay for the 429 branch to the Retry-After header value, with a configurable cap to bound worst-case waits, and only fall back to a default delay when the header is absent
• •After verifying the parser handles both delta-seconds and HTTP-date forms, add a unit test that asserts the policy waits at least the header value before the next attempt
• •Disable layered retries one at a time and re-run the controlled repro; keep the configuration that honors the header and remove the layer that does not, to eliminate competing retry decisions
• •Where a scheduler or queue consumer reissues work, gate the re-enqueue on the maximum Retry-After observed for the batch, so the cadence cannot undercut the header
• •Condition any change on the proof step below; do not ship a sleep-based workaround before the parser and policy have been verified end to end

Prove the fix

1. 01Run a controlled repro against a stub server that returns 429 with a known Retry-After value and verify that the next outbound request is observed no earlier than that value, with a small tolerance for jitter
2. 02Inspect the retry policy write path and confirm in code review that the 429 branch resolves its delay from the header, not from a constant
3. 03Capture a production trace for the previously failing endpoint and confirm the inter-request gap between the 429 response and the next attempt is greater than or equal to the Retry-After value the server returned
4. 04Remove the stub and re-enable the real provider; confirm in the next incident that the Retry-After header is logged, parsed, and respected by the active retry policy

Prevention and next steps

• •Treat Retry-After as a first-class field in the retry policy schema; lint or schema-check policies so a 429 branch without a header-driven delay fails review
• •Document a single retry layer per service in the architecture overview, and require an ADR when a second layer is introduced
• •Add an integration test that asserts the client waits at least the header value before retrying, and run it in CI against a stub server for each provider integration
• •Alert on the ratio of 429 responses to subsequent requests within the header window; a sustained non-zero ratio is an early signal that the window is being ignored

Safe commands and checks

grep -rn "Retry-After" <service-source-root>
grep -rn "retry_after\|retryAfter\|RETRY_AFTER" <service-source-root>
grep -rn "status_code.*429\|on_429\|whenStatus(429" <service-source-root>
grep -rn "sleep(\|backoff(\|exponential" <service-source-root>