API 502 upstream reset: correlate gateway errors with connection teardown

The symptoms

• •Gateway returns HTTP 502 to the client with a generic body, while the upstream service logs no corresponding successful or failed application request for that correlation identifier.
• •502 frequency rises in lockstep with upstream TCP connection churn, indicated by rising connection-reset or connection-closed counts at the proxy.
• •Latency histogram shows a bimodal pattern: a fast-fail cluster at the gateway timeout floor and a normal cluster for successful responses, with no slow middle band.
• •Upstream process metrics show normal CPU and memory, but socket or file descriptor counts spike before each 502 burst, suggesting active connection teardown.
• •Gateway access logs show the response was generated by the proxy itself with no upstream response headers recorded, distinguishing a teardown from an upstream 5xx that was relayed.

Likely causes

• •Upstream process recycles or restarts mid-request, closing its listening sockets and forcing in-flight TCP connections to terminate before any HTTP response is written.
• •Reverse proxy or load balancer idle timeout, keepalive timeout, or request timeout fires before the upstream finishes producing a response, causing the proxy to abort the connection.
• •Operating system or container resource limit reached (file descriptors, ephemeral ports, connection tracking table) causing the kernel to reset new or existing sockets.
• •Network path between gateway and upstream drops the connection due to firewall state table expiry, NAT timeout, or an intermediate device sending TCP RST during reconfiguration.
• •Upstream application bug or runtime signal handling that closes the socket on uncaught exception, panic, or shutdown hook without writing a status line.

First ten minutes

1. 01Confirm the 502 is emitted by the gateway itself by checking that the gateway access log entry contains no upstream_status field and no upstream response headers, only the proxy-generated 502.
2. 02Extract the gateway-emitted correlation or request identifier and search the upstream service logs for any record with that identifier, including accepted and rejected access lines, to determine whether the upstream ever saw the request.
3. 03Capture gateway-side connection metrics: count of TCP RST received from upstream, count of connections closed by upstream before any response, and gateway-side request timeouts for the same window.
4. 04Compare the 502 timestamps against upstream deployment, restart, configuration reload, autoscaling, or drain events to test whether teardown aligns with upstream lifecycle changes.
5. 05Check upstream host or pod resource saturation indicators: open socket count, file descriptor count, ephemeral port availability, and any container cgroup limit reached events.
6. 06Review recent changes to gateway timeout, keepalive, and pool settings versus the upstream's documented or observed response time distribution to identify a timeout-induced teardown.

Evidence to collect

• •Gateway access log line for the failing request including request id, upstream address attempted, bytes sent to upstream, bytes received from upstream, and total latency.
• •Upstream access or application log entries keyed by the same request id, including any connection-accepted and connection-closed events from the upstream side.
• •Gateway-side counter snapshot for the incident window: 502 responses, upstream connect errors, upstream RSTs, and upstream timeouts, broken down by upstream backend instance.
• •Upstream host metrics for the same window: open connections, accepted connections, closed connections, file descriptor usage, CPU, and memory pressure.
• •Network device or firewall state table statistics if the path traverses a known stateful middlebox, focusing on drops and resets in the incident window.

Where to look

• •The gateway-to-upstream boundary: proxy access logs, upstream connection pool state, and gateway-emitted error codes versus relayed upstream codes.
• •The upstream service accept and close paths: application server connection lifecycle logs, runtime socket stats, and any connection-limit or backpressure warnings.
• •The host or container running the upstream: kernel socket accounting, file descriptor limits, ephemeral port range, and cgroup or namespace restrictions.
• •The network path between gateway and upstream: stateful devices such as load balancers, NAT gateways, and firewalls that may inject RSTs or drop idle entries.
• •Configuration and deployment systems: recent restarts, rolling deploys, autoscaling thresholds, and configuration reload events on both gateway and upstream.

Diagnostic steps

1. 01Classify the 502 origin by inspecting the gateway log for absence of an upstream_status field and absence of any response header received from upstream; if present, the gateway merely relayed an upstream 5xx rather than experiencing teardown.
2. 02Test whether the upstream ever observed the request by joining the gateway request id against upstream logs; absence of any record suggests teardown before the upstream finished reading the request or before it logged acceptance.
3. 03Measure the time from gateway connect-to-upstream to gateway emit-502 against the configured gateway read or request timeout to determine whether timeout, rather than RST, is the proximate cause.
4. 04Correlate 502 bursts with upstream lifecycle events by overlaying restart, deploy, and drain timestamps on the 502 timeline; sustained baseline 502 with no lifecycle event points to timeout or resource limit instead of restart.
5. 05Compare per-upstream-instance 502 rates against per-instance resource metrics to isolate a single bad instance, which suggests host or container limits rather than a uniform network cause.
6. 06Differentiate a client-side RST propagating to the gateway from an upstream RST by inspecting gateway logs for the direction of close: a reset while the gateway is still sending the request body indicates upstream or network teardown, not client cancellation.

Common mistakes

• •Treating every 502 as the upstream's application returning an error, when the gateway can emit 502 itself when the upstream connection terminates before a status line is received, as defined for the 502 status.
• •Assuming client request cancellation explains all connection resets seen at the gateway; upstream-initiated resets and network-path resets can produce identical socket-level signals but require different fixes.
• •Increasing gateway timeouts without first measuring upstream response time distribution, which can mask a real upstream stall rather than addressing the teardown root cause.
• •Reloading upstream configuration or restarting upstream pods during diagnosis, which destroys the in-flight connection state evidence needed to confirm the teardown source.
• •Conflating an upstream HTTP 5xx relayed by the gateway with a gateway-generated 502 caused by teardown, leading teams to fix application code when the proxy boundary is the actual failure surface.

Safe fixes

• •If the gateway 502 timestamps align with upstream restart or deploy events, enable connection draining on the upstream and reduce the gateway's keepalive or pool idle timeout below the upstream's drain window so new requests land on warmed instances.
• •If the gateway latency-to-502 equals the configured gateway timeout, raise the timeout only after confirming upstream p99 response time fits inside the new value, and add a circuit breaker so repeated timeouts stop sending traffic rather than silently extending waits.
• •If upstream file descriptor or socket limits are saturated, raise the limit to a value justified by observed peak open connections plus headroom, and add monitoring that alerts before the limit is reached rather than after resets begin.
• •If the path traverses a stateful middlebox with idle timeouts shorter than the gateway's keepalive, shorten the gateway's TCP keepalive interval to match, or move to a path that does not traverse the device, so the middlebox does not inject resets.
• •If a single upstream instance shows elevated 502s while others are healthy, remove it from rotation after capturing its diagnostics, and replace rather than restart in place so the failing state can be inspected offline.

Prove the fix

1. 01Re-run the same client workload that produced the original 502 burst and observe that the gateway emits 0 entries with no upstream_status and no upstream response headers over a window at least as long as the original incident.
2. 02Confirm upstream logs now contain an accepted-request record for a statistically significant sample of gateway request ids that previously had no upstream record, indicating the upstream completed the TCP and HTTP exchange.
3. 03Verify the gateway counter for upstream connection resets and connection-closed-before-response remains at its pre-incident baseline during a synthetic load test that previously reproduced the failure.
4. 04Check that the distribution of upstream response times remains inside the configured gateway timeout with margin, so that a return to elevated latency cannot silently reintroduce timeout-driven 502s.
5. 05Observe that upstream resource headroom for sockets, file descriptors, and ephemeral ports remains above the alert threshold during peak load, so a saturation-induced reset cannot recur without an early warning.

Prevention and next steps

• •Instrument the gateway boundary to record per-request whether an upstream response was received and whether the upstream connection was reset, and alert on sustained deviation from baseline rather than only on 5xx totals.
• •Keep gateway timeout, keepalive, and pool idle settings versioned and reviewed whenever upstream response time distribution shifts, so timeout-induced teardown cannot drift in unnoticed.
• •Require upstream deploys and autoscaling events to drain connections rather than hard-restart listeners, and align gateway pool settings with the documented drain duration.
• •Track upstream socket, file descriptor, and ephemeral port utilization as first-class capacity metrics with alerts set well below hard limits.
• •Maintain a documented mapping of stateful middleboxes on the gateway-to-upstream path along with their idle timeout values, and keep TCP keepalive below the smallest such value on every network segment in use.

Safe commands and checks

grep -n 'upstream_status' /var/log/gateway/access.log | awk '$0=="-" {c++} END{print "gateway_generated_502:", c+0}' — counts gateway access log lines where upstream_status is missing, indicating the 502 was emitted by the proxy rather than relayed from upstream.
grep '<request_id>' /var/log/gateway/access.log — locates the single gateway access log line for the failing request to read bytes sent to and received from upstream and the recorded latency.
grep '<request_id>' /var/log/upstream/access.log — searches the upstream access log for the same identifier to determine whether the upstream ever recorded the request.
ss -s — prints kernel socket summary including sockets in use and time-wait count, useful for confirming saturation of the local socket table on the upstream host.
ss -ant | awk 'NR>1 {print $1}' | sort | uniq -c — tallies TCP socket states on a host so a spike in CLOSE-WAIT or TIME-WAIT can be matched against the incident window.
cat /proc/<pid>/limits | grep -E 'open files|max user processes' — inspects the file descriptor and process limits of the upstream process whose pid is obtained from the upstream service manager, without modifying anything.