API timeout diagnosis checklist

The symptoms

• •Client logs show a request that began within budget but never received a response, headers, or a final status code, while the same call succeeds on a retry or from a different network path.
• •An intermediate proxy or gateway reports an upstream connect or read deadline that elapsed before any HTTP response status was emitted, often surfacing as a 502 with an empty or abbreviated error body.
• •An upstream service records that the request arrived but was aborted or cancelled mid-flight, frequently correlated with a client-side abort signal or socket close rather than a 5xx response it generated.
• •Observability shows tail latency above the configured deadline while the median latency remains low, indicating a small subset of requests exhaust capacity rather than a uniform slowdown.
• •Retry amplification is visible: client retry counters increase while origin request counts stay flat or fall, implying the original requests are still in flight when the timeout fires.

Likely causes

• •Client-side deadline configured shorter than the realistic end-to-end budget required for the upstream service under current load, including serialization, connection setup, and TLS handshake.
• •Reverse proxy or load balancer connect, read, or inactivity timeouts that are tighter than the upstream's typical processing time for this endpoint, producing 502 Bad Gateway responses when the upstream is still working.
• •Upstream service processing time exceeding the deadline due to synchronous blocking work, lock contention, cold caches, or dependency calls that themselves have their own timeouts to consider.
• •Network-layer stalls: TCP retransmits, TLS renegotiation, DNS resolution delays, or middlebox-induced resets that hold the socket open without bytes flowing until a read deadline expires.
• •Resource exhaustion on the origin: thread pool saturation, connection pool exhaustion, or event loop backpressure that delays the moment the upstream begins emitting a response.
• •Application-level cancellation, where the client framework aborts a still-pending request on navigation, user action, or session termination, observed by the origin as a truncated or never-finished request.

First ten minutes

1. 01Identify the boundary that fired the timeout: classify the event as client-fired, proxy-fired, or origin-fired by inspecting the actor and event type in the nearest log, then record which deadline value was exceeded.
2. 02Capture the request's observed start time, last byte received time, and timeout value at each boundary, and compute the gap between deadline and actual elapsed time to determine how close the failure was to the configured limit.
3. 03Pull the matching trace or correlation identifier from the closest boundary and follow it through the chain; if no identifier exists, freeze a sample request and add one before continuing so subsequent boundaries can be aligned.
4. 04Compare the timeout's failure rate against the endpoint's p50, p95, and p99 latency and against the upstream's processing-time distribution to decide whether the timeout is a budget problem or a tail problem.

Evidence to collect

• •Per-boundary log entries naming the deadline that expired, the request identifier, the connection or socket identifier, and whether a response status had been emitted before the timer fired.
• •Distributed trace spans for client, proxy, and origin segments, with explicit timer annotations indicating wait, connect, TLS, request-send, response-wait, and response-read phases.
• •Endpoint latency histograms from the origin for the same time window, broken down by status and route, to compare the timeout value against the actual processing-time tail.
• •Proxy or gateway access logs distinguishing connection timeouts from read timeouts and from upstream connect failures, since each implies a different layer of the request lifecycle.
• •Resource indicators at the origin during the incident window: thread pool depth, connection pool occupancy, event loop lag, and any dependency call latency that the request was waiting on.

Where to look

• •The client boundary: framework HTTP client timeout configuration, request cancellation hooks, and any user-driven abort signals that close the socket before the server responds.
• •The edge and proxy boundary: reverse proxy and load balancer settings for connect, send, read, and keep-alive timeouts, and any request buffering that changes when bytes actually traverse the proxy.
• •The origin boundary: web server and application server request processing timers, including time-to-first-byte, handler execution time, and any asynchronous work that completes after the initial response.
• •The dependency boundary: outbound calls from the origin to databases, caches, and downstream services, each with its own deadline that compounds the end-to-end budget.
• •The network boundary: TLS handshake time, DNS resolution time, TCP retransmit counters, and any middlebox or service mesh sidecar that introduces its own timer.

Diagnostic steps

1. 01Bound the failure by reading the actor in the log: a client framework timeout message implies a client boundary fault, while a proxy 502 with no upstream status implies a proxy boundary fault, and an origin log entry showing a truncated request implies an origin boundary fault.
2. 02Quantify the budget gap by computing deadline minus observed elapsed time at the failing boundary; values near zero indicate a budget mismatch, while values far from zero indicate a real stall or hang.
3. 03Decompose latency along the trace into wait, connect, TLS, send, first-byte, and last-byte phases to localize which phase consumed the budget, since each phase has a distinct remediation.
4. 04Test the hypothesis with a controlled replay: a same-payload request issued at low load from a fixed client will either succeed quickly, succeed slowly, or fail at the same boundary, which distinguishes capacity problems from configuration problems.
5. 05Cross-check the origin's processing-time tail against the failing deadline using histograms from the incident window to determine whether the origin is simply slower than the budget permits.
6. 06Inspect upstream dependency timers in the trace to determine whether the origin's latency is its own work or waiting on another service's deadline, which changes whether the fix belongs to this service or its dependency.
7. 07Verify retry behavior by counting client retries against origin request volume; a wide gap confirms that originals are still pending when timeouts fire, which informs whether retries are amplifying the problem.

Common mistakes

• •Assuming the proxy or gateway is the root cause when the timeout value was simply smaller than the realistic upstream latency, and shortening the proxy deadline further as a misguided mitigation.
• •Increasing the timeout unconditionally without first measuring which phase is exhausted, which can mask a hang by extending the window in which a stuck request holds resources.
• •Adding retries without jitter or a circuit breaker, which converts a small tail-latency problem into a thundering herd that worsens the underlying capacity issue.
• •Treating a 502 Bad Gateway as a definitive upstream fault without checking whether the proxy emitted it because its own read or connect deadline expired while the upstream was still processing the request.
• •Reading client-side stack traces as evidence of server failure when they actually originate from a client cancellation or a too-aggressive client deadline that the upstream never had a chance to satisfy.

Safe fixes

• •If the client deadline is shorter than the upstream's observed p99 processing time, raise the client deadline to a value informed by the trace's phase breakdown and re-measure, keeping the new value at or above the proxy's deadline to avoid shifting the failure point.
• •If the proxy read deadline is tighter than the origin's p99 plus network round-trip budget, align the proxy's connect, send, and read timers to the origin's documented SLO and verify with a replay that 502s attributable to timeouts cease.
• •If origin processing time consistently exceeds the agreed budget, address the cause inside the origin rather than lengthening deadlines, by offloading blocking work, raising concurrency limits, or terminating unbounded waits on dependencies with explicit deadlines.
• •If retries are amplifying the incident, add exponential backoff with jitter, a per-host retry cap, and a circuit breaker that opens on sustained timeout rates before re-enabling automatic retry.
• •If dependency timers compound the budget, replace nested synchronous waits with explicit per-dependency deadlines that sum to less than the end-to-end budget, and surface dependency-induced waits as a distinct metric.

Prove the fix

1. 01Confirm that for the previously failing endpoint, the timeout failure rate over a representative load window drops to within the agreed error budget and the p99 latency no longer clusters at the former deadline value.
2. 02Confirm via distributed traces that the failing phase now completes within its phase budget and that the request's elapsed time is bounded by the slowest legitimate phase rather than by a deadline expiration.
3. 03Confirm that retry counters at the client boundary match the volume of new requests at the origin, indicating that originals are not being abandoned mid-flight and re-issued as duplicates.
4. 04Confirm via a controlled replay at low and at peak load that the same payload now completes or fails with a definitive application status rather than with a deadline-fired event at any boundary.
5. 05Confirm that a synthetic canary issuing the same call on a fixed interval reports a stable success rate and that its end-to-end latency distribution sits below the lowest configured deadline in the chain.

Prevention and next steps

• •Document an explicit end-to-end deadline for each endpoint that names every boundary in the path, including client, proxy, and origin, with each boundary's timer set no tighter than the documented budget.
• •Emit a structured timeout event at every boundary that includes the request identifier, the deadline value, the elapsed time, and the phase that exhausted the budget, so that future incidents can be classified without guesswork.
• •Track per-endpoint latency histograms and alert on the gap between the configured deadline and the observed p99, so that drift between configuration and reality is visible before users observe failures.
• •Standardize retry policy across clients with jitter, a maximum attempt count, and a circuit breaker, and verify in load tests that retries do not convert tail latency into an outage.

Safe commands and checks

echo "Inspect a structured timeout event: confirm fields request_id, boundary, deadline_ms, elapsed_ms, phase, response_status_emitted"
grep -n "boundary=" <path-to-timeout-log> | head -n 50
awk -F',' '$3 == "client" {count++} $3 == "proxy" {count++} $3 == "origin" {count++} END {for (k in count) print k, count[k]}' <path-to-timeout-log>
awk -F',' 'NR>1 {print $4 - $5}' <path-to-timeout-log> | sort -n | awk '{a[NR]=$1} END {print "p50", a[int(NR*0.50)], "p95", a[int(NR*0.95)], "p99", a[int(NR*0.99)]}'
grep -n "deadline_ms=" <path-to-trace-export> | awk -F'deadline_ms=' '{print $2}' | awk -F' ' '{print $1}' | sort -n | uniq -c | sort -rn | head -n 10
grep -n "phase=" <path-to-trace-export> | sed -E 's/.*phase=([^,]+).*/\1/' | sort | uniq -c | sort -rn
awk -F',' '$6 == "false" {print $1, $2, $3}' <path-to-timeout-log> | head -n 20