API returns 429 only during bursts: calculate the request shape

The symptoms

• •429 Too Many Requests responses appear only when traffic ramps up quickly, even though average requests-per-second stays below the documented quota.
• •Steady-state calls succeed for hours, then a deploy, retry storm, scheduled job, or cron fan-out produces a wave of 429s that clears once the burst subsides.
• •The 429 responses cluster at the start of the burst and taper off as backoff kicks in, suggesting a burst window rather than a sustained rate cap.
• •Retry-After or X-RateLimit-Reset headers are present on 429 responses but absent on successful responses, indicating a quota boundary is being crossed.
• •Errors surface in only one client region, worker pool, or downstream consumer, while other callers with the same API key remain unaffected.
• •The HTTP status code 429 appears alongside 200s in the same second, which rules out a hard outage and points to a per-window quota.

Likely causes

• •Burst rate exceeds the provider's token bucket capacity even though the long-run average stays under the documented per-second limit.
• •Connection or concurrency cap is reached: the client opens more parallel requests during a burst than the provider allows per credential or per IP.
• •Retry storms amplify an initial transient error into a burst that breaches the rate policy, often after a 502 or 503 from the same provider.
• •Time-window misalignment: the provider's window boundary does not align with the client's measurement, so a burst that spans two windows can be charged against a smaller remaining budget.
• •Per-endpoint limits lower than the global quota: a specific route has its own ceiling that is exceeded only when the burst targets that endpoint.
• •Shared credential fan-out: multiple internal callers share one API key, and their bursts happen to coincide during a deploy or scheduled job.

First ten minutes

1. 01Capture the exact timestamp window in which 429s appeared, and the request count per second during that window, from the access log or API gateway telemetry.
2. 02Pull the response headers from the 429 responses, especially Retry-After, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, or provider-specific equivalents.
3. 03Compare the burst peak (max requests in a 1-second sliding window) against the documented per-second and burst limits in the provider's API reference.
4. 04Check the client-side concurrency at the moment of the burst: how many in-flight requests were open, and from how many distinct workers or pods.
5. 05Look for a preceding 5xx or network error in the same minute that may have triggered retry amplification.
6. 06Identify whether all 429s share one API key, one source IP, or one downstream service, which would localize the quota being hit.

Evidence to collect

• •Access log entries showing HTTP status, request timestamp with millisecond precision, response time, and the response headers listed above.
• •Client-side metrics for in-flight request count, queue depth, and worker concurrency at the burst window.
• •Provider documentation page citing the exact per-second, per-minute, burst, and concurrency limits for the endpoint in question.
• •Deployment or job schedule timeline showing whether the burst coincides with a release, cron, or batch run.
• •Trace or correlation IDs from the 429 responses to determine whether retries from a single upstream request are responsible.

Where to look

• •The API gateway or reverse proxy access log, where status code, response headers, and upstream service are recorded per request.
• •The HTTP client or SDK telemetry layer, which records in-flight counts, retry counts, and per-request timing.
• •The provider's developer documentation, specifically the rate limiting, authentication, and quotas sections for the called endpoint.
• •The orchestrator or scheduler logs (Kubernetes events, cron logs, queue depth metrics) for the burst trigger event.
• •The shared credential store or secrets manager, to confirm whether multiple services route through one key.
• •The retry policy configuration in the client library or service mesh, where backoff multiplier and max attempts live.

Diagnostic steps

1. 01Build a 1-second histogram of request counts over the burst window and identify the peak value, which is the candidate burst rate.
2. 02Read the 429 response headers and note the values of any rate-limit header fields; compare the indicated reset time to the observed end of the burst.
3. 03Cross-reference the peak rate and concurrent in-flight count against the provider's documented burst ceiling and concurrency ceiling separately, because one may bind before the other.
4. 04Reproduce the burst shape against a staging credential or sandbox endpoint, then confirm whether 429s reappear at the same peak; this isolates the cause to request shape rather than data or payload size.
5. 05Disable retries temporarily and replay the original burst; if 429s disappear, retry amplification was contributing to the breach.
6. 06Inspect the distribution of source identifiers (API key, IP, pod) across the 429 responses to determine whether the quota is per-credential, per-IP, or global.
7. 07Check whether the 429s target a single endpoint or are spread across routes, which distinguishes per-endpoint quotas from account-wide limits.

Common mistakes

• •Assuming the average requests-per-second is the binding constraint, when the provider enforces a separate burst or concurrency ceiling.
• •Reading Retry-After as a fixed backoff for all clients, when it reflects only the remaining window for the specific key that was throttled.
• •Conflating client-side rate limiting with provider-side rate limiting; a client limiter that is set above the provider ceiling will appear to work until a burst occurs.
• •Reducing concurrency without reducing peak rate, which can lower in-flight counts but leave the burst ceiling breached because requests still arrive in the same 1-second window.
• •Adding retries with exponential backoff that still fires inside the same window as the original burst, amplifying rather than smoothing the load.
• •Sharing one API key across many services without coordinating their burst windows, so individual services look compliant while the aggregate breaches.

Safe fixes

• •If peak 1-second rate exceeds the documented burst ceiling, add a token bucket or leaky bucket on the client that caps the peak to the documented value, and verify with a replay test.
• •If concurrent in-flight requests exceed the concurrency limit, introduce a bounded semaphore or connection pool whose size matches the documented ceiling, then re-measure during a burst.
• •If retry amplification is contributing, switch to jittered exponential backoff with a floor that pushes retries past the current rate-limit window, and cap max attempts.
• •If multiple services share one credential, partition the key or move to per-service credentials and coordinate their burst windows through a shared scheduler.
• •If a per-endpoint quota is the binding constraint, route lower-priority traffic to a secondary credential or a cached response layer to keep the burst under the per-route ceiling.
• •If the provider offers a quota increase, request it only after documenting the measured peak and the business need, so the new ceiling matches the actual burst shape.

Prove the fix

1. 01Replay the original burst pattern against the same endpoint and credential, and confirm that the peak 1-second request count matches the burst window but produces zero 429 responses.
2. 02Observe the X-RateLimit-Remaining or equivalent header during the replay; it should stay above zero throughout the burst and only drop at the documented ceiling.
3. 03Run a synthetic burst test in CI that fires the same peak rate for 60 seconds and asserts that no response carries a 429 status, providing a regression guard.
4. 04Confirm that the Retry-After header is not consumed by the client during normal operation, by inspecting the access log for any 429 entries during the verification window.
5. 05Verify that concurrent in-flight requests stay at or below the configured semaphore size during the burst, by sampling the client-side gauge at the peak.

Prevention and next steps

• •Document the provider's burst, concurrency, and per-endpoint limits alongside the API key in the secrets manager, so every consumer reads the same ceiling.
• •Add a load test to CI that exercises the production burst pattern, including retry behavior, and fails the build on any 429 in the synthetic run.
• •Centralize the rate-limiting client or SDK so all callers share one token bucket view rather than each enforcing limits independently.
• •Alert on a leading indicator such as X-RateLimit-Remaining dropping below a threshold, before the quota is breached and 429s appear.
• •Coordinate deploys, crons, and batch jobs that target the same provider so their burst windows do not stack into a single 1-second peak.

Safe commands and checks

awk '{print substr($1,1,19)}' access.log | sort | uniq -c | sort -rn | head -20
grep -E ' 429 ' access.log | head -50
grep -iE 'retry-after|x-ratelimit-(limit|remaining|reset)' access.log | sort | uniq -c | sort -rn | head -20
awk '/ 429 /{print $0}' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -10