How to verify API timeout cancellation releases resources

The symptoms

• •Client times out (configurable read deadline) yet the server-side request handler continues to run to completion and writes a response that is discarded by the client.
• •Open file descriptors, sockets, or stream handles held by the timed-out request grow monotonically across successive timed-out calls; pool metrics do not return to baseline within one timeout interval.
• •A 502 Bad Gateway status code appears on retry, indicating the upstream was already in a degraded state when the timeout fired, consistent with resources not being released after a prior timeout.
• •Memory or coroutine/thread count for the service plateaus at a higher level after a burst of timeouts than after a burst of successful requests of equal size.
• •Cancellation tokens or abort signals are observed being created on timeout but never observed transitioning to a cancelled state in traces or logs.

Likely causes

• •The handler ignores the framework-supplied cancellation token or abort signal and awaits a non-cancelable primitive (raw socket read, third-party SDK call without a context argument, synchronous blocking call).
• •Downstream dependencies (database driver, gRPC client, message broker producer) are invoked without forwarding the cancellation token, so they continue holding connections after the HTTP layer has given up.
• •Response body streaming is started but never closed on timeout; the write half stays open until the upstream produces the full payload, leaking a stream handle per timed-out request.
• •Timeout is implemented at the edge (reverse proxy, ingress, client SDK) only, while the application code has its own internal deadline that is longer or absent, decoupling client-visible timeout from server-side work cancellation.
• •Resource pools (DB, HTTP client to upstream, worker queue) do not validate liveness of returned items on release, so a timed-out handle is returned to the pool in a half-closed state and corrupts the next caller.
• •Async runtime cancellation propagates only at await points; CPU-bound or tight-loop work between await points cannot be interrupted, so work continues past the timeout deadline even though the token is cancelled.

First ten minutes

1. 01Capture the exact timeout configuration on each hop (client SDK, ingress, service handler, downstream client) and write down which component owns the deadline that fires first.
2. 02Reproduce a single timed-out request against a non-production target and record the response status, the time the client gave up, and the time the server log shows the handler returning.
3. 03Pull the per-process handle counters (open sockets, FDs, active streams, goroutines/threads) before the request, immediately after timeout, and one full timeout interval later; classify the gap as leaked or reclaimed.
4. 04Grep service logs for the request ID and look for both the timeout event and any subsequent completion event; two completion-shaped entries for one request ID is a strong cancellation-failure signal.
5. 05Inspect the response headers actually returned for an indication that the server was aware the client had disconnected, versus silently finishing the payload.
6. 06Decide before deeper work whether the timeout is fired by the client, the ingress, or the service itself; the verification procedure differs for each owner.

Evidence to collect

• •Per-request trace showing the cancel/abort event timestamp relative to the timeout deadline and the handler return timestamp.
• •Handle or FD count sampled at three points: pre-request, post-timeout, and after one timeout interval of idle.
• •Downstream dependency telemetry (DB connection checkout time, broker producer ack time) for the same request ID, to confirm whether the dependency observed cancellation.
• •Pool checkout and check-in events paired by handle ID, to detect returns that occur after the parent request has already timed out.
• •Reverse-proxy access log entry for the same request ID, including status code and bytes sent, to distinguish a clean 502 with no body from a fully streamed 200 the client dropped.

Where to look

• •The HTTP server framework's request lifecycle: middleware that registers cancellation callbacks, and the point at which the response writer is closed.
• •The boundary between the service handler and each downstream client (database driver, HTTP client, gRPC stub, message producer) where the cancellation token must be threaded through.
• •The reverse proxy or ingress layer that may emit its own timeout and surface a 502 Bad Gateway before the upstream has actually freed resources.
• •The runtime scheduler view (goroutine dump, thread dump, async task list) where work that survived a timeout will appear as a still-running unit tied to the old request ID.
• •Resource pool internals: the release path that returns a connection or handle when the request ends, and whether it consults a liveness check before reuse.

Diagnostic steps

1. 01Compare the handler's observed end time to the client-observed timeout; if handler-end is later by more than one timeout interval, the handler did not honor cancellation.
2. 02Search the handler source for any call site that accepts a cancellation token, abort signal, or context, and confirm each downstream call forwards it; treat missing forwarding as a primary cause.
3. 03Identify the exact await or blocking primitive that runs past the deadline; classify it as cancelable (respects the token), partially cancelable (cancels only at the next poll), or non-cancelable (must be wrapped or aborted via a separate channel).
4. 04Differentiate a true cancellation failure from an ingress-originated 502 by checking whether the upstream emitted a partial response body before the gateway gave up.
5. 05Validate pool hygiene by issuing one timed-out request and then one immediate successful request; if the second request fails with a transport or EOF error, the timed-out handle was returned dirty.
6. 06Confirm runtime scheduling cooperates with cancellation by checking whether any non-async code path executes between await points during the timed-out interval.

Common mistakes

• •Conflating client-side socket closure with server-side work cancellation; the client can give up while the server continues, and only the server side proves resource release.
• •Trusting a 502 status code as proof that the upstream cleaned up; a 502 from an ingress or gateway does not describe the state of upstream resources, only of the edge.
• •Adding a global timeout in the reverse proxy and assuming the application handler also stops; layered timeouts create gaps where work runs with no one listening for the result.
• •Returning a connection to a pool in a finally or defer block without first checking whether the underlying transport is still open, which leaks a half-closed handle into the next caller.
• •Measuring resource leaks only by peak count and missing the post-timeout plateau, which is where unreleased work most clearly accumulates relative to baseline.
• •Forgetting that CPU-bound work between await points cannot be cancelled by the runtime and will continue to hold any resource it has acquired.

Safe fixes

• •If evidence shows the handler runs past the deadline, thread the framework's cancellation token into every downstream call inside that handler, then verify with a per-request trace that each dependency receives a cancel event within one network round-trip of the timeout.
• •If evidence shows stream or response body is left open, add an explicit close on the cancellation path and confirm via metrics that the stream count drops at the timeout timestamp rather than at response completion.
• •If evidence shows pool corruption after timeouts, introduce a liveness probe on release so that any handle whose underlying transport is closed is destroyed rather than returned to the pool.
• •If evidence shows non-cancelable work blocks release, restructure the hot path so the blocking primitive runs in a child unit whose result is awaited with the token, and short-circuit the await on cancellation.
• •If evidence shows ingress-originated 502s mask real cleanup, align ingress timeout to be strictly longer than the service timeout so the service always observes its own deadline first and can release before the edge intervenes.
• •Each fix must be paired with the proof step below before being considered resolved; do not ship a change that only changes status codes or logs without changing the resource counters.

Prove the fix

1. 01Run a controlled burst of timed-out requests against the target service and confirm that the FD, socket, goroutine, or stream counter returns to within a small tolerance of its pre-burst baseline within one configured timeout interval after the last request.
2. 02For each request ID in the burst, confirm via trace that the handler end time is at or before the timeout timestamp and that all downstream dependencies logged a cancellation event for the same ID.
3. 03Follow each timed-out request with one normal request on the same upstream dependency and confirm the normal request succeeds without transport errors, proving the pool was not contaminated.
4. 04Capture a regression check that fails the build if the post-burst counter exceeds baseline by more than the configured tolerance, so future changes cannot silently re-introduce the leak.
5. 05Verify that under a sustained load of timed-out requests the service does not produce 502 Bad Gateway responses attributable to its own resource exhaustion, separating edge-side 502s from service-side ones.

Prevention and next steps

• •Adopt a single owner of the timeout deadline per request hop and document it; do not allow independent timeouts on the same request without an explicit precedence rule.
• •Make cancellation token forwarding a lint or static-check rule for every new outbound client call added to a request handler.
• •Track a resource-leak regression metric (counter returned to baseline within one timeout interval after a timeout burst) and alert on its drift.
• •Require that any new blocking primitive in a request path be either inherently cancellation-aware or wrapped behind an await that the runtime can interrupt.
• •Periodically inject timeouts in pre-production and compare handler end times to timeout timestamps; treat divergence as a release blocker.

Safe commands and checks

ss -tan state time-wait-recv '( sport = :<port> )' | wc -l   # count half-open sockets on the service port after a timeout burst; replace <port> with the listener port obtained from the service configuration.
ls /proc/<pid>/fd | wc -l   # count open file descriptors for the service process; replace <pid> with the service PID and sample before, immediately after, and one timeout interval after a controlled timeout burst.
grep -n 'request.cancelled\|context.cancel\|abort.signal' <service_log_path>   # search the service log for cancellation events tied to request IDs involved in the timeout burst; replace <service_log_path> with the configured log file path.
grep -E 'request_id=<request_id>\b' <access_log_path> | awk '{print $4, $7, $9}'   # inspect the ingress or reverse-proxy access log for the request ID; replace <request_id> and <access_log_path> with values from the trace to compare upstream versus edge timestamps.