How to verify an API recovers after an upstream reset

The symptoms

• •Initial failure surfaces as HTTP 502 from the gateway or proxy, but the origin returns 2xx on retry, indicating the upstream socket was torn down mid-flight rather than the application failing.
• •Clients see inconsistent outcomes for the same idempotency key: some attempts report 502, others return success, suggesting the reset happened after the server committed a side effect.
• •Logs show the gateway marking the upstream as "unhealthy" and draining the connection pool, while the application server logs show no application-level error for the same request id.
• •After the reset, traffic shifts to a fallback origin or region and recovers within seconds, but auditors flag duplicate downstream notifications for a subset of reset-window requests.

Likely causes

• •The gateway observed a TCP RST or abrupt FIN from the upstream before it could read a complete response header, which the spec classifies as an upstream connectivity failure rather than an application error.
• •The client retried without consulting an idempotency record, so a request that already mutated state on the origin was replayed, producing duplicate side effects.
• •The gateway's reset detection conflated socket-level reset with application-level 5xx, marking the upstream unhealthy too aggressively and propagating 502 instead of waiting for the next probe.
• •Upstream keepalive timeout or connection pool churn exceeded the proxy's read timeout, so the gateway closed the socket and the origin interpreted the close as a cancellation.

First ten minutes

1. 01Classify the event as a reset, not an application error: confirm the gateway emitted 502 while the origin's access log for the same request id shows no completed handler or shows a partial write before disconnect.
2. 02Capture the exact request id, idempotency key, and timestamp from both the gateway access log and the origin access log so every later decision can be tied to a single in-flight request.
3. 03Freeze retry behavior: switch the client or gateway to a known-safe retry posture that records attempts, so subsequent verification can distinguish replay from fresh execution.
4. 04Take a read-only snapshot of connection-pool counters and upstream health state from the gateway control plane to establish the pre-recovery baseline.
5. 05Decide the recovery boundary: identify whether verification will be performed against the same origin after it reaccepts connections, a warm fallback origin, or a degraded-response mode, and record that decision.

Evidence to collect

• •Gateway access log lines tagged 502 with the upstream address, connection-close reason, and bytes-read-before-disconnect, matched by request id to origin logs.
• •Origin access log lines for the same request id showing either no handler entry or a partial response write, to distinguish a reset from a completed 2xx.
• •Idempotency store records indicating whether the request id was committed before the reset, including any replay timestamps from client retries.
• •Connection-pool telemetry: open connections, idle connections, failed handshakes, and the timestamp at which the upstream was marked healthy again.

Where to look

• •At the gateway boundary, in the proxy's access and error logs and in its upstream health-check state, since that is where 502 is generated and where reset classification is recorded.
• •At the origin boundary, in the application server's access log and any per-request trace spans, to determine whether the handler ran to completion or was interrupted by socket close.
• •At the client boundary, in the HTTP client's retry layer and idempotency cache, where duplicate side effects would be observable as repeated mutation requests with the same key.
• •At the network boundary, in the load balancer or sidecar's connection table, where keepalive timeouts, idle reaps, and TCP RST events are first observable.

Diagnostic steps

1. 01Map every 502 emitted during the incident window to its origin handler outcome; a handler that completed indicates the reset occurred after side-effect commit, which is the case that demands idempotent recovery.
2. 02Compare client retry attempts against the idempotency store: a retry that bypasses the store, or a store that does not record the key until after commit, predicts duplicate side effects on recovery.
3. 03Inspect the gateway's reset-to-unhealthy mapping: if a single RST marks the upstream down for the full pool, the system will over-report 502 and starve the recovery path.
4. 04Reproduce the reset in a controlled read-only probe by sending a request with a known idempotency key, observing whether the origin completes the handler before any socket teardown, and checking that the gateway's response carries the expected status once the upstream is marked healthy.
5. 05Trace the post-reset recovery path: confirm the gateway drains the affected connection, opens a fresh socket, and that the first request on the new socket returns the same status a steady-state request would, with no cached 502.

Common mistakes

• •Treating every 502 as an application bug; the spec defines 502 as a gateway-side classification for an upstream connectivity failure, so an origin returning 2xx after the reset does not contradict the gateway's earlier 502.
• •Retrying on the client without consulting an idempotency record, which converts a recoverable reset into a duplicate side-effect incident the moment the origin becomes reachable again.
• •Marking the upstream unhealthy for the whole pool on a single reset, which inflates the 502 window and obscures whether recovery actually occurred versus being short-circuited by the gateway.
• •Verifying recovery by hitting a health endpoint only; a green health probe does not prove that the request path will not replay a previously committed side effect under retry.

Safe fixes

• •Conditional on evidence that the origin committed the side effect before the reset: require the client to consult the idempotency store before retry, and require the store to record the key at handler entry, not at handler exit, so the retry can be short-circuited.
• •Conditional on evidence that the gateway over-classified resets as pool-wide failures: scope the unhealthy marking to the single connection or upstream instance, and require a successful fresh-socket probe before re-admitting traffic to that target.
• •Conditional on evidence that the reset window produced duplicate downstream effects: introduce a deduplication layer keyed by request id and idempotency key at the egress boundary, and verify it only when the origin log confirms prior commit.
• •Conditional on evidence that keepalive churn caused the reset: align the gateway's read timeout with the origin's request timeout plus a documented margin, and confirm the alignment by inspecting the recorded bytes-read-before-disconnect in the gateway log.

Prove the fix

1. 01Re-run the controlled read-only probe with a fresh idempotency key: a forced socket reset on the first attempt must produce 502 from the gateway, the retry must consult the idempotency store, and the origin must execute the handler exactly once with no duplicate downstream notification.
2. 02Replay a request that already has a stored idempotency record: the client must observe a cached or replayed response and must not emit a second mutation to the origin, confirming the retry path is idempotent rather than best-effort.
3. 03Observe the gateway's connection pool across one reset event: the affected upstream must be marked unhealthy only for the failed connection, the next request on a new socket must return the steady-state status, and no further 502 must be emitted for the same root cause within the recovery window.
4. 04Audit the post-fix incident window for duplicate side-effect records under any request id that previously hit 502, and confirm the duplicate count is zero once the idempotency check is in place.

Prevention and next steps

• •Standardize on request-id propagation from client to gateway to origin so every retry, reset, and recovery decision can be correlated to a single in-flight request rather than reconstructed from timestamps.
• •Treat reset detection as a first-class signal at the gateway, distinct from application 5xx, with its own scoped unhealthy state and its own probe-before-admit policy.
• •Make idempotency a precondition for any retry policy on mutating endpoints, with the store write happening before the handler commits so that a reset cannot create a replayable gap.
• •Periodically rehearse the recovery protocol in a non-production environment using a forced socket reset, and require the rehearsal evidence to include both the gateway 502 and the origin's single execution to be considered passing.

Safe commands and checks

echo "Inspect gateway access log lines tagged 502 for the incident window, filtered by request id, and record upstream address, connection-close reason, and bytes-read-before-disconnect for each line."
echo "Inspect origin access log lines for the same request id and record handler entry, response status, and bytes-written to distinguish a reset before commit from a reset after commit."
echo "Query the idempotency store for the request id and idempotency key captured in step 1, and record whether the key was present before the reset, committed during the handler, or missing entirely."
echo "Read the gateway control-plane snapshot for the upstream pool: open connections, idle connections, failed handshakes, and the timestamp at which the upstream transitioned from unhealthy to healthy, taken before and after the recovery probe."
echo "Run a controlled read-only recovery probe with a fresh idempotency key by issuing a single request that exercises the documented reset path, then verify the gateway returned 502 on the first attempt and the origin executed the handler exactly once on the retry, with no duplicate downstream notification."
echo "Replay a request whose idempotency key is already stored, and verify the client received the stored response without emitting a second mutation to the origin, confirming the retry path is idempotent."
echo "After the probe, audit the post-fix window for any duplicate side-effect record under a request id that previously produced 502, and confirm the duplicate count is zero."