502 incident debugging checklist

The symptoms

• •Clients receive 502 Bad Gateway responses from the proxy or load balancer instead of successful or upstream-authored errors.
• •Upstream origin returns 200 directly when probed, while the fronting proxy still returns 502 to the same path.
• •Intermittent 502s correlate with connection resets, TLS handshake failures, or read timeouts in proxy access logs.
• •Error rate spike coincides with a deployment, upstream restart, autoscaling event, or certificate change rather than uniform traffic shift.

Likely causes

• •Upstream is reachable on the network but returns malformed, partial, or non-HTTP bytes that the proxy cannot parse as a valid response.
• •TLS or HTTP/2 negotiation fails between proxy and origin because of expired certificates, SNI mismatch, or incompatible protocol versions.
• •Proxy upstream pool is exhausted: keep-alive connections saturated, worker queue full, or active health checks failing so the pool is marked empty.
• •DNS or routing inconsistency: proxy resolves a stale origin IP, or the origin IP changes after an autoscaler event and the proxy has not refreshed.

First ten minutes

1. 01Capture the exact 502 response line, headers, and correlation ID from the proxy access log; note timestamp, client IP, requested path, and upstream pool or backend name.
2. 02Confirm whether the 502 is uniform across paths and clients, or scoped to one route, region, upstream pool, or client segment.
3. 03Check proxy health check status for each upstream pool and read the most recent health check failure reason before changing configuration.
4. 04Issue one read-only HTTP request to the upstream origin bypassing the proxy using the production host header; compare status code and response size to the proxied response.
5. 05Look for recent change events in the same window: deploys, upstream restarts, certificate rotations, DNS changes, autoscaling thresholds, or WAF rule updates.
6. 06Decide scope before mitigation: route-specific fix, pool isolation, or full incident; record the decision in the incident channel before any config change.

Evidence to collect

• •Proxy access log entry showing the 502 with upstream address, connect time, response time, and the proxy's error or disconnect reason code.
• •Proxy error log entries for the same correlation ID, including TLS handshake errors, read timeouts, connection refused, or empty response flags.
• •Upstream origin log or metric for the same request window: request count, response codes, active connections, and any restart or rotation event.
• •Health check probe results and the timestamp of the last successful probe for the failing upstream pool.
• •Recent configuration or topology diff: deploy SHA, DNS record TTL, certificate serial, autoscaling min/max, and proxy upstream block.

Where to look

• •At the proxy boundary: access and error logs, upstream pool state, active connection count, and health check subsystem output.
• •At the origin boundary: origin process logs, listen socket state, upstream response headers as received by the proxy, and recent container or VM restart timestamps.
• •On the network path between proxy and origin: DNS resolution cache on the proxy, TCP connect attempts, TLS handshake details, and any intermediate load balancer or sidecar.
• •In the change ledger: deployment records, certificate and SNI rotations, DNS record edits, autoscaling rule changes, and proxy configuration reloads in the incident window.

Diagnostic steps

1. 01Classify the 502 by proxy reason code if logged: connect failure, read timeout, empty response, invalid response, or upstream protocol error; each points to a different subsystem.
2. 02Compare the origin's direct HTTP response to the proxied response on the same path and host header; mismatch confirms the proxy is the failure boundary.
3. 03Trace the proxy-to-origin TCP and TLS handshake for a single failing request and compare to a passing request to isolate protocol or routing divergence.
4. 04Inspect the proxy's upstream pool health and last successful probe timestamp; a recently failed probe explains pool draining but does not by itself prove root cause.
5. 05Check DNS resolution used by the proxy for the upstream hostname against the authoritative record; record TTL and last cache time.
6. 06Review upstream resource signals: connection saturation, listener backlog, CPU, memory, and thread or worker pool exhaustion in the failing window.
7. 07Correlate the first 502 timestamp with the most recent change event; if no change event aligns, broaden evidence collection rather than guessing.

Common mistakes

• •Restarting the proxy or upstream without first classifying the 502 reason code, which can mask intermittent causes such as pool exhaustion or stale DNS.
• •Treating 502 as an application bug because the upstream returns a clean response when tested in isolation; the proxy boundary must still be proven.
• •Replacing certificates or rotating keys when the actual failure is a connect timeout or read timeout rather than a TLS alert.
• •Flushing DNS caches globally before confirming that the proxy is using a stale origin record; flushing client caches does not affect the proxy.
• •Increasing upstream pool size without evidence of saturation, which can mask a more serious protocol or routing defect.

Safe fixes

• •If proxy health checks are failing and origin is reachable, temporarily isolate the failing pool from the load balancer and route traffic to the surviving pool while investigating.
• •If DNS for the upstream is stale on the proxy, trigger a controlled upstream pool reload or DNS refresh on the proxy only, after confirming the authoritative record is correct.
• •If the origin is returning malformed responses under load, reduce proxy-to-origin concurrency or enable or tighten request queuing limits to stabilize the proxy pool.
• •If TLS negotiation is failing, align SNI and certificate SAN entries on the origin and confirm supported protocol versions before disabling older versions.
• •Hold any configuration change until the proxy reason code and origin evidence are recorded; revert candidate changes one at a time to keep regression attribution possible.

Prove the fix

1. 01Proxy access logs show zero new 502 responses on the affected route and pool for at least two consecutive health check intervals after the change.
2. 02Proxy upstream pool reports all probes passing and the last successful probe timestamp continues to advance without reset.
3. 03An end-to-end synthetic request through the proxy returns the same status code and response body as a direct origin request on the same path and host header.
4. 04Origin connection and error metrics for the proxy client return to pre-incident baseline with no new disconnect or handshake error counts.
5. 05The change that resolved the incident is documented with timestamp, author, scope, and the evidence entry that motivated it, and is left in place until a follow-up review.

Prevention and next steps

• •Define and alert on proxy 502 rate, upstream pool probe failure rate, and origin connect and read timeout counts with per-route and per-pool dimensions.
• •Track DNS TTL and certificate expiry for every upstream hostname, and rehearse proxy reload and DNS refresh procedures in staging.
• •Keep the proxy upstream block under version control with peer review, and require an incident reference for any emergency change made outside review.
• •Run scheduled end-to-end probes through the proxy to a known origin path so proxy-to-origin regressions are detected before customer impact.

Safe commands and checks

grep -n " 502 " <proxy_access_log> | tail -n 50
awk '$9 == "502" {print $4, $7, $9, $11}' <proxy_access_log> | sort | uniq -c | sort -rn | head
grep -E "upstream|connect|handshake|timeout" <proxy_error_log> | tail -n 100
dig +short <upstream_hostname> @<authoritative_resolver>
ss -tan state established '( dport = :<upstream_port> or sport = :<upstream_port> )' | head
openssl s_client -connect <upstream_hostname>:<upstream_port> -servername <upstream_hostname> -brief < /dev/null
curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' -H 'Host: <production_host>' https://<upstream_hostname><request_path>