API intermittently returns 502: trace the proxy boundary

The symptoms

• •Same endpoint returns 502 for some requests and 200/4xx for others, with no code change between attempts, indicating a non-deterministic boundary rather than an application logic bug.
• •502 rate correlates with concurrency, payload size, or request latency rather than with a specific user, route, or time-of-day window, suggesting a pool or timeout interaction at the gateway.
• •502 responses arrive with a proxy-supplied header (e.g., Server, Via, or X-Cache) while the upstream's own error pages and traces are absent, indicating the response was constructed by the intermediary.
• •Retry of the identical request body and headers succeeds within seconds, ruling out request-shape defect as the sole cause and pointing to a transient proxy-to-upstream condition.
• •Backend access logs show no entry for the failed request, confirming the request did not reach the application layer and the failure occurred before upstream logging.
• •Some 502 responses carry a connection-reset or empty-body signal at the client while the proxy logs a different outcome, a split-brain symptom common at TLS or keep-alive boundaries.

Likely causes

• •Upstream socket closed mid-response: the origin terminated the TCP connection before sending a complete, parseable response, often due to upstream worker recycling or a stale keep-alive connection in the proxy pool.
• •Upstream timeout exceeded: the proxy's connect, read, or send timeout fired before headers were received, and the proxy surfaced 502 instead of 504 because the configured class treats timeout-class failures as gateway errors.
• •Protocol mismatch at the boundary: HTTP/1.1 vs HTTP/2 frontend talking to an HTTP/1.1 backend, or an X-Forwarded-* / Host header rewrite that the upstream rejects, causing it to reset the connection.
• •Upstream resource exhaustion: origin process pool, file descriptor limit, or accept queue saturated, so a fraction of new connections are accepted then immediately closed, producing a 502 at the proxy.
• •DNS or routing flap between proxy and origin: the proxy resolves a name, attempts a connection, and the upstream address is stale or rotating, producing a transient connect failure that the proxy reports as 502.

First ten minutes

1. 01Confirm the 502 is emitted by the proxy and not the origin: inspect the response headers for a proxy-owned Server or Via field, and compare the response body signature to the origin's known error page; a proxy-constructed body strengthens the boundary hypothesis.
2. 02Capture the failing request's exact method, path, headers, and body size, then replicate from the same client network using an identical request envelope; success on retry confirms intermittency rather than a malformed request.
3. 03Locate the proxy access log and filter for the same minute and client IP; record the upstream-status, upstream-address, request-time, and upstream-connect-time fields the proxy emits, since each differentiates a cause.
4. 04Locate the upstream access and error logs on the origin host; absence of any entry for the failing request proves the request never reached the application, isolating the failure to the proxy↔origin segment.
5. 05Check the proxy's connection-pool state and current upstream latency percentiles; elevated p99 with normal p50 indicates tail-latency-driven connect or read timeouts at the boundary.
6. 06Correlate the 502 timestamps with any recent origin deploy, restart, scaling event, or DNS change, since each can briefly invalidate the proxy's upstream pool.

Evidence to collect

• •Proxy access log line for the failing request, including the upstream-status, upstream-address, request-time, and upstream-connect-time fields, kept verbatim for cross-host correlation.
• •Response headers from the failing request, specifically Server, Via, X-Cache, X-Request-ID, and any proxy-injected correlation header, to identify the responder and enable log join.
• •Upstream access log search for the same correlation ID and time window, with the result (hit or miss) recorded as the primary boundary-pass or boundary-fail signal.
• •TCP-level evidence from the proxy host: socket-level reset, premature close, or TLS handshake failure observed for the failing upstream connection, ideally with a short packet capture window.
• •Origin process and resource snapshot at the failure timestamp: open file descriptors, accept queue depth, worker count, and current connection count, to test the exhaustion hypothesis.
• •Distribution of 502s by upstream-address and by upstream port, to test whether a single pool member or a single route is responsible rather than the entire upstream.

Where to look

• •The proxy↔origin boundary, specifically the proxy's upstream connection pool, keep-alive timer, and per-route health check, since 502 is the proxy's verdict on a pool member's response.
• •The upstream accept queue and worker entry point, because a saturated accept queue causes the proxy to receive a connection it cannot use, which the proxy relays as 502.
• •The DNS resolution path between proxy and origin, including any TTL-driven change, because a stale or rotating A record can produce connect failures that surface as 502 for a subset of requests.
• •The TLS termination point on the proxy, since a protocol downgrade, SNI mismatch, or expired upstream certificate causes the proxy to abandon the connection and emit 502.
• •The application framework's request entry on the origin, including any middleware that strips or rewrites headers, because a header-rewrite mismatch can cause the origin to close the connection mid-response.

Diagnostic steps

1. 01Classify each 502 by proxy-emitted upstream status: empty, zero, or a 5xx that the origin never logged, because each value eliminates a different layer (no log entry rules out the application; nonzero status with a log entry shifts suspicion to the response-rewrite path).
2. 02Compute the 502 fraction per upstream pool member; a single member elevated above the rest implicates that host, while a flat distribution implicates a shared resource such as accept queue or DNS.
3. 03Compare request-time to upstream-connect-time plus upstream-response-time on the failing lines; if request-time greatly exceeds the sum, the proxy spent time in its own queue or buffer, pointing to proxy-side saturation rather than upstream latency.
4. 04Test the protocol hypothesis by checking the proxy's configured upstream protocol against the origin's listener; a mismatch in HTTP/1.1 vs HTTP/2 or in expected ALPN produces intermittent resets that the proxy reports as 502.
5. 05Test the stale-connection hypothesis by toggling the proxy's upstream keep-alive or by routing a probe through a fresh pool member; if 502s disappear for the probe, the pool is recycling connections the origin is closing.
6. 06Test the timeout hypothesis by raising the connect and read timeouts in a non-production route and observing whether the 502 fraction drops; a drop confirms timeout-class failure, while no change points to connection-reset class.
7. 07Test the resource-exhaustion hypothesis on the origin by sampling accept queue depth, file descriptor count, and worker saturation at the failure timestamps; saturation coinciding with 502 supports the upstream-exhaustion cause.
8. 08Test the DNS hypothesis by resolving the upstream hostname from the proxy host at the failure timestamp and comparing the answer to the proxy's currently connected upstream address; divergence implicates a stale or rotating resolver.

Common mistakes

• •Assuming 502 means the origin application crashed; in practice 502 is the proxy's report, and the origin may be healthy while the proxy's pool member is bad, so always cross-check the origin's own logs before changing application code.
• •Confusing 502 with 504; 504 is a gateway timeout with a stricter semantic, and treating them identically leads to misconfigured timeouts that mask the real cause, so classify the proxy's upstream-status field before tuning.
• •Increasing timeouts globally without evidence; a blanket timeout raise can mask resource exhaustion and shift the failure from 502 to slow 200s, so only raise timeouts after the diagnostics confirm a timeout-class failure.
• •Restarting the proxy as a first response; this clears the pool and any keep-alive state, which can temporarily hide a stale-connection bug rather than fix it, so reproduce the 502 on a fresh pool before accepting a restart as proof.
• •Ignoring the empty-body case; a 502 with empty body and no upstream-status field often signals a TCP-level reset, not an HTTP-level failure, so the diagnostic must include socket-level evidence rather than only HTTP traces.
• •Looking only at the client-side error; the proxy and the origin each have partial evidence, and a single-side view cannot distinguish proxy-side from origin-side failure, so evidence must be collected from both segments.

Safe fixes

• •If diagnostics confirm a stale-connection cause, reduce the proxy's upstream keep-alive idle timeout to a value below the origin's own idle close, and enable proactive connection recycling on the proxy pool member, then verify by re-running the same request mix.
• •If diagnostics confirm a timeout-class cause, raise only the specific timeout (connect, read, or send) that the proxy log identifies as the bottleneck, scoped to a single route, and only to a value supported by the origin's SLO, then verify by re-measuring the 502 fraction.
• •If diagnostics confirm an upstream-exhaustion cause, increase the origin's accept queue, file descriptor limit, or worker pool by an amount justified by the observed saturation, and verify that 502s drop without a corresponding increase in origin 5xx.
• •If diagnostics confirm a DNS flap, pin the proxy to a specific upstream address via a stable resolver or shorten the resolver's TTL handling, and verify by re-running the request mix against the same hostname from the proxy host.
• •If diagnostics confirm a protocol mismatch, configure the proxy's upstream protocol to match the origin's listener exactly, and verify by capturing the next 502 case and confirming the upstream-status field is no longer empty.
• •If no cause is confirmed, do not change production configuration; instead, deploy a canary with telemetry that exposes the same proxy log fields at higher fidelity, and continue evidence collection until a cause is classified.

Prove the fix

1. 01The 502 response fraction for the affected route, measured over at least one full traffic cycle, returns to the pre-incident baseline, and the proxy's upstream-status field for the same request shape is no longer empty or zero.
2. 02An identical replicated request, issued through the same client and proxy path, returns a non-502 response within one second, and a second identical request issued in the same second also returns a non-502 response, ruling out a single-retry artifact.
3. 03The upstream access log now contains an entry for the previously-missing request correlation ID, confirming the request now crosses the proxy↔origin boundary instead of being terminated by the proxy.
4. 04Socket-level evidence shows no premature close or TLS handshake failure for the proxy↔origin connection during the verification window, eliminating the reset-class failure for the verified route.
5. 05A canary or staging route carrying the same configuration does not regress over a defined observation window, and the proxy's upstream-connect-time and upstream-response-time percentiles remain within the documented SLO.

Prevention and next steps

• •Emit a structured proxy access log line for every request that includes the upstream-status, upstream-address, request-time, upstream-connect-time, and a correlation ID, so that future 502 investigations can join proxy and origin logs without guesswork.
• •Set proxy upstream keep-alive and timeout values from the origin's own SLO rather than from defaults, and review them whenever the origin changes its listener, protocol, or worker model.
• •Monitor the 502 fraction per upstream pool member and per route, with an alert threshold tied to a baseline plus a small tolerance, so that a single bad pool member is detected before it broadens into a global symptom.
• •Keep the origin's accept queue, file descriptor limit, and worker pool sized against the proxy's maximum concurrent connections, and re-evaluate the sizing whenever the proxy's pool size changes.
• •Reserve a canary route that mirrors production configuration and exposes the same proxy log fields, so that future configuration changes can be validated for 502 regression before they reach general traffic.

Safe commands and checks

grep -F "<correlation-id>" /var/log/proxy/access.log | awk '{print $0}' | head -n 5  # print the proxy access log lines for a correlation ID; replace <correlation-id> with the value from the failing response header.
grep -F "<correlation-id>" /var/log/origin/access.log | wc -l  # count origin-side entries for the same correlation ID; zero confirms the request did not reach the application.
ss -tan state time-wait | awk '{print $4}' | cut -d: -f2 | sort | uniq -c | sort -rn | head  # list external source ports in TIME_WAIT on the proxy host; a single dominant port indicates a client retry pattern rather than a pool issue.
ss -s  # print socket summary on the proxy host, as a coarse check for socket-table pressure that correlates with proxy-emitted 502s.
getent hosts <upstream-hostname>  # resolve the upstream hostname from the proxy host using the system resolver; run at the failure timestamp and compare to the proxy's currently connected upstream address.
openssl s_client -connect <upstream-host>:<upstream-port> -servername <upstream-hostname> -brief </dev/null  # probe the upstream TLS endpoint from the proxy host; replace placeholders with the observed upstream host, port, and SNI; a handshake failure points to the TLS boundary.
awk '$9 == "502" {print $0}' /var/log/proxy/access.log | awk '{print $14}' | sort | uniq -c | sort -rn  # group 502 lines by upstream address field to test whether a single pool member is responsible; field index depends on the proxy log format.
awk '$9 == "502" {print $NF}' /var/log/proxy/access.log | sort | uniq -c  # inspect the last field of 502 lines, typically the upstream-status reported by the proxy, to classify between empty, zero, and nonzero upstream responses.