API 429 diagnosis checklist

The symptoms

• •Application logs repeatedly emit HTTP 429 responses from an upstream or peer service while the client-side request volume has not visibly changed.
• •User-facing operations fail intermittently but correlate with request bursts, deployment rollouts, batch jobs, or a newly added service consumer.
• •Errors disappear shortly after a retry-with-backoff, with no underlying data or dependency state change, suggesting client pace rather than service health.
• •Retry-After header is absent on some 429 responses and present on others, indicating that different gateways or middlewares are enforcing the limit.
• •Monitored traffic shows a sharp edge at a fixed requests-per-window boundary, with success before the boundary and uniform 429 after it.

Likely causes

• •A request quota tied to an API key, tenant, or user identity has been reached because legitimate traffic grew past the configured allowance.
• •A burst allowance or token bucket has emptied faster than the refill rate when concurrent clients overlap, even though the average rate looks compliant.
• •A reverse proxy, API gateway, or WAF is applying a global throttle that is independent of the upstream service's own quota.
• •Retry amplification: a downstream timeout or 5xx triggered client retries that pushed aggregate request volume across the limit.
• •Shared quota pools: multiple services or environments are using the same API key, so one consumer's burst exhausts the budget for the others.
• •Misconfiguration: a rate limit value or window size was changed in configuration, an environment was promoted with stale cache values, or a canary was routed with doubled traffic share.

First ten minutes

1. 01Confirm the boundary: read three recent 429 response lines from the application or proxy log and record the path, client identity (API key id, tenant id, IP), and exact timestamp; uniform rejection across paths suggests a gateway, path-specific rejection suggests the origin service.
2. 02Inspect headers on the 429 response: note whether Retry-After, the IETF draft RateLimit-Limit / RateLimit-Remaining / RateLimit-Reset, or X-RateLimit-* headers are present, since their values indicate whether the server intends for you to retry and when the window resets.
3. 03Compare volume to budget: compute requests-per-minute for the affected identity over the last 5 minutes against any documented or observed limit, and check whether the boundary coincides with the 429 onset.
4. 04Rule out collateral damage: scan for concurrent 5xx, timeout, or connection-reset events from the same dependency that could explain retry storms; a 429 that follows a wave of 503s points to amplification, not steady-state throttling.
5. 05Snapshot scope: list every service, job, and environment that shares the API key or identity currently being throttled so you know whether isolation is possible before any policy change.

Evidence to collect

• •Three or more raw HTTP 429 response samples including all headers and response body, captured with the originating request method, path, and client identity.
• •Request-rate time series for the affected identity, broken down by service or consumer, aligned to the minute the first 429 appeared.
• •Documentation or configuration values for the rate limit policy in effect: requests allowed, window length, burst size, and which dimension the limit is keyed on (IP, API key, tenant, route).
• •Recent change record: configuration diffs, deployment markers, traffic-routing changes, or new consumers added in the 24 hours preceding the first 429.
• •Retry-After and RateLimit-* header values on both 429 and successful responses, used to verify window reset behavior end-to-end.

Where to look

• •The origin service's access log for the path returning 429, filtered by the client identity that is being throttled, to distinguish origin-enforced from gateway-enforced rejection.
• •The API gateway, ingress, or reverse proxy layer in front of the service, since most quotas are enforced before the request reaches the origin and leave a distinct log line.
• •Any WAF, CDN edge, or DDoS-protection tier that may apply its own per-IP throttle independent of application-level limits.
• •Quota or billing dashboards for the upstream provider, which may expose a finer-grained view of which identity is consuming budget and over which window.
• •Client-side retry configuration and libraries, including jitter, maximum attempts, and circuit breakers, to determine whether client behavior is amplifying the throttle.

Diagnostic steps

1. 01Classify the enforcer: send an identical request from two distinct clients (different API key or IP) within the same second; if both receive 429, the limit is keyed on a broader dimension such as route or IP rather than identity, and the diagnosis shifts to infrastructure rather than quota.
2. 02Measure the window: with a steady, known request rate, observe at what rate the 429s begin and stop; the on/off pattern defines the window length and request budget, which you can compare against published limits.
3. 03Test Retry-After fidelity: honor the Retry-After value exactly once after a 429 and confirm that the next request succeeds; if it still returns 429, the window has not reset and the header is unreliable, which changes the fix strategy.
4. 04Isolate the consumer: route or disable the suspected consumer and observe whether the 429 rate for other consumers drops; a drop confirms shared quota, a flat rate suggests the throttle is path-keyed.
5. 05Cross-check upstream timeouts: pair the 429 timeline with any 5xx or timeout spikes; if 429s follow 5xx, treat the throttle as a symptom of retry amplification and address the upstream fault, not just the rate.
6. 06Verify configuration resolution: if a config change is suspected, confirm that the running process loaded the new value by reading the effective setting from the gateway control plane or a documented introspection endpoint, never from memory alone.

Common mistakes

• •Immediately raising the limit or requesting a quota increase without first proving that legitimate traffic is the cause; this can mask a retry storm and inflate cost.
• •Implementing client-side exponential backoff that ignores Retry-After, which causes immediate retry pile-ups exactly when the window has not yet refilled.
• •Treating the 429 as a transient failure inside a circuit breaker, so that the client stops calling entirely instead of slowing down, masking whether the upstream has recovered.
• •Assuming every gateway layer honors the same headers and applies the same key; an edge WAF may throttle by IP while the origin throttles by API key, producing inconsistent 429 samples from a single client.
• •Sharing one API key across production, staging, and CI, so a test run exhausts the budget for live users and the resulting 429 is misattributed to a code regression.
• •Editing rate-limit values in a dashboard without redeploying or invalidating cached config, which leaves the old limit in force and produces the false conclusion that the change did not take effect.
• •Chasing the 429 by adding more parallelism in the client, which deepens the throttle and can trigger secondary limits such as concurrent connection caps.

Safe fixes

• •Introduce client-side pacing that honors Retry-After first, then applies jittered exponential backoff up to a documented cap; gate the change on a feature flag so you can disable it within minutes if it masks another fault.
• •Separate API keys or identities per environment and per service consumer, verified by inspecting the key id on a sampled 429 response, so a noisy consumer cannot exhaust the budget of others.
• •Add a token-bucket or leaky-bucket regulator at the client when burstiness, not steady-state rate, is the root cause; size it from the observed window and budget, not from a guess.
• •Place a server-side guard that converts repeated 429s on the same upstream into a single, deduplicated request with a cached fallback, to keep user-facing latency bounded while you address the underlying quota.
• •If the 429 follows a wave of 5xx or timeouts, fix the upstream fault first and add a brief client retry pause before resuming traffic, rather than raising the limit while the upstream is degraded.
• •If the quota is genuinely too small for legitimate traffic, request an increase only after attaching evidence: the time series, identity breakdown, and a projection of new load at the requested ceiling.

Prove the fix

1. 01Run a synthetic client for one full window length at the prior peak rate plus a defined margin, and observe zero 429s in the gateway and origin logs for that identity, with RateLimit-Remaining or equivalent never reaching zero.
2. 02Confirm that honoring Retry-After yields a 2xx on the immediately following request, repeated across at least three independent windows, demonstrating the header is authoritative.
3. 03Inject a controlled burst above the limit and verify the system returns 429 with Retry-After, not 5xx, and that non-bursted consumers continue to succeed, proving isolation between identities.
4. 04Replay a captured sequence of 5xx followed by retries and observe that the client now produces no additional 429s beyond the unavoidable ones, confirming retry amplification has been removed.
5. 05Compare per-identity request rates before and after the change; the previously throttled identity remains at or below the documented budget for 24 hours with no 429s, while other identities are unaffected.
6. 06Review alerts: the rate of 429s for the affected identity drops to zero, and no new 429s appear for other identities sharing the same gateway or key pool.

Prevention and next steps

• •Publish the active rate limit policy per identity and per environment in a shared, version-controlled location, and require any change to reference an incident or capacity review.
• •Issue distinct API keys per service and per environment, validated by a deployment check that fails the rollout if a forbidden key appears in the configuration.
• •Instrument every client with structured logs of Retry-After values and observed response codes, and alert on a sustained non-zero 429 ratio per identity, not on raw traffic volume.
• •Exercise rate-limit behavior in pre-production using a synthetic load test that exceeds the documented budget, so the team's first encounter with a 429 is in a controlled setting.
• •Maintain a runbook that lists the enforcer boundary, the shared quota map, and the rollback path for any client pacing or key-rotation change, reviewed at least once per quarter.

Safe commands and checks

grep -n ' 429 ' <access-log-path> | tail -n 50
awk '{print $4}' <access-log-path> | sort | uniq -c | sort -nr | head -n 20
grep -E 'Retry-After|RateLimit-Limit|RateLimit-Remaining|RateLimit-Reset|X-RateLimit' <captured-429-response-path>
journalctl -u <service-unit> --since '<timestamp>' --until '<timestamp>' | grep -i '429'
kubectl logs <pod-name> --since=10m | grep -E '429|rate.?limit'