API times out only for large payloads: locate upload and processing deadlines

The symptoms

• •Small requests under an empirical size threshold (often tens of KB to low MB) succeed, while requests above it fail with timeouts, 502, or 504 codes.
• •Failure latency is stable and close to a fixed number of seconds, suggesting a wall-clock deadline rather than a crash.
• •Failure rate correlates with Content-Length or request duration rather than with error rate per endpoint path.
• •Server-side logs show partial processing (connection accepted, request started, body partially read) but no completion record, or show an upstream disconnect mid-upload.
• •Retries with the same payload sometimes succeed after a brief pause, pointing to a transient deadline rather than a structural bug.

Likely causes

• •An edge or proxy upload deadline (for example, a body-read timeout) that is shorter than the time needed to stream the buffered payload, so the upstream closes the connection before the application finishes reading.
• •An application or framework read timeout that fires while the body is still being parsed or buffered into memory, with no streaming path configured.
• •A reverse proxy buffer limit being exceeded, causing the request to be rejected or held until another timer expires.
• •A downstream service or worker that performs synchronous, CPU- or memory-bound work (parsing, transcoding, validation, third-party call) whose cost scales with size and crosses a synchronous deadline.
• •Client-side or middleware write timeout that disconnects before the server has acknowledged receipt, producing an aborted upload mistaken for a server timeout.

First ten minutes

1. 01Confirm the correlation between payload size and failure by graphing response code or latency against Content-Length on the last 24 hours of traffic.
2. 02Identify the exact threshold by scanning for the smallest failing payload and the largest succeeding payload, then computing the size and time gap.
3. 03Capture the response status line, headers, and server timing if exposed, and look for evidence of an upstream-vs-origin split (different status depending on hop).
4. 04Pull the corresponding server log line for one failing request and one succeeding request, aligned by request id, and compare which hop ended the request.
5. 05Decide whether the connection was closed during upload (read-side deadline) or after the request was fully received but before response (processing deadline).

Evidence to collect

• •Distribution of response status codes binned by Content-Length bucket, to show the size threshold numerically.
• •Server-side timing fields such as time-to-first-byte, request processing duration, and upstream connect time, when emitted by the proxy or framework.
• •Proxy and load balancer logs indicating which component closed the connection and at what stage of the request lifecycle.
• •Application logs showing whether the request body was fully received, buffered, or only partially read before the timeout fired.
• •Resource metrics (CPU, memory, event-loop lag, file descriptor counts) captured during a failing window for one representative large payload.

Where to look

• •At the edge boundary: the TLS terminator or CDN layer, where request body acceptance and read timers are configured separately from response timers.
• •At the proxy boundary: the reverse proxy or ingress, where buffering, body size limits, and per-stage timeouts are defined and logged.
• •At the application boundary: the framework or server runtime, where body parsing, deserialization, and request-scoped limits are enforced.
• •At the workload boundary: the synchronous handler or worker queue, where per-request deadlines and downstream call budgets are tracked.
• •At the client boundary: the HTTP client or SDK, where write, connect, and overall request timeouts can abort the upload prematurely.

Diagnostic steps

1. 01Re-issue a failing request with a measured payload and record the precise failure latency; if the latency clusters near a fixed value, the cause is a deadline rather than capacity.
2. 02Compare the failure latency against the documented or observed timeout values on each hop to identify which timer is most likely expiring first.
3. 03Reduce payload size incrementally toward the threshold and re-test to localize the inflection point, which indicates whether a fixed-size limit or a cost-scaling issue is dominant.
4. 04Inspect the proxy log for the stage at which the connection was closed: before the body was fully read, after reading but before forwarding, or after forwarding but before the response.
5. 05Enable verbose framework logging for one request to record whether body parsing completed and what downstream call was in flight when the deadline fired.
6. 06Cross-check resource metrics during the failure window to rule out a secondary capacity cause that simply correlates with larger payloads.
7. 07Re-run the same test from a controlled client with explicit, longer write and overall timeouts to determine whether the client itself is closing the upload early.

Common mistakes

• •Assuming the application handler is the default suspect without checking the upstream read-side timer, which is the most common cause of size-correlated failures.
• •Chasing a server-side code path instead of inspecting whether the request body was ever fully received, leading to misallocated debugging effort.
• •Increasing a single timeout value uniformly, which often masks the failing hop while leaving the actual deadline unchanged and produces different failure modes later.
• •Treating intermittent retries as confirmation of a transient network issue, when in fact the retry succeeded because the deadline was not yet exhausted.
• •Optimizing payload format or compression before establishing that the boundary causing the failure is purely CPU-bound rather than time-bound.

Safe fixes

• •If the proxy log shows the connection closed during body read, raise the body-read timer on the consuming hop only by enough to cover the 95th-percentile large-payload transfer time, and verify with a controlled re-test.
• •If the failure occurs after full receipt but before response, set the downstream call budget explicitly and stream large bodies to a worker rather than processing them synchronously in the request path.
• •If a buffer limit is being exceeded, switch the affected hop from full buffering to streaming mode so memory usage no longer scales with the size of a single request.
• •If the client is closing the upload early, configure the client with separate connect, write, and read timeouts and ensure the write timeout exceeds the expected upload duration.
• •Each fix above must be accompanied by a regression check that re-issues the largest previously failing payload and observes a successful response within the new deadline.

Prove the fix

1. 01The 95th-percentile large-payload request now returns a 2xx response and the failure bucket for that size range is empty across a one-hour observation window.
2. 02Server logs for the same payload show the request body fully received and the handler completing within the configured deadline, with the terminating hop matching the intended configuration.
3. 03A regression test that posts a payload just above the historical failure threshold succeeds and continues to succeed across at least ten consecutive runs without timeout or 502.
4. 04Latency for the previously failing size bucket is now bounded by an explicit, documented deadline rather than by an observed failure wall, and resource metrics show no correlated spike.

Prevention and next steps

• •Maintain a documented map of timeout and buffer budgets per hop, with each value justified against the 95th-percentile payload size for that boundary.
• •Alert on failure rate per Content-Length bucket rather than only on global error rate, so size-correlated regressions are detected before user impact.
• •Include a synthetic large-payload probe in the standard test suite, with an explicit pass criterion that the probe completes within the documented deadline.
• •Prefer streaming or asynchronous processing for endpoints whose payloads are unbounded, and encode that decision in the endpoint design rather than relying on per-request tuning.

Safe commands and checks

awk '{print $9, $10}' access.log | awk '{ if ($2 ~ /^[0-9]+$/) { size=$2; code=$1; if (size >= 1048576) bucket=">=1MB"; else if (size >= 65536) bucket=">=64KB"; else bucket="<64KB"; count[bucket, code]++ } } END { for (k in count) print k, count[k] }'
grep -n 'request-id' reverse-proxy.log | awk -F'request-id=' '{print $2}' | awk '{print $1}' | sort -u | head -n 20
awk '$7 ~ /50[24]/ {print $4, $7, $9, $NF}' proxy.log | sort -k1,1 | head -n 50
awk '{ if ($NF ~ /timeout/) print $4, $7, $9, $NF }' reverse-proxy.log | sort -k1,1 | head -n 50
ps -o pid,etime,pcpu,pmem,comm -p <pid>
ss -ltnp 2>/dev/null | awk '{print $4, $6}' | head -n 40