API connection pool leak: identify requests that do not release clients

The symptoms

• •Pool saturation error such as "ECONNRESET after socket timeout," "TimeoutError: waiting for connection from pool," or "Pool is closed" appearing intermittently under steady traffic
• •Successful requests stop while CPU and event loop are idle, indicating waiter threads or microtasks are blocked on a checkout queue rather than CPU work
• •Steadily rising count of in-use sockets, checked-out clients, or active handles, with a non-zero value even when the request rate is zero, proving release is not happening
• •Heap or resident memory grows in proportion to in-flight requests, since unreleased clients retain sockets and response buffers
• •Error path leaves clients checked out: requests that throw, reject, or have their response body aborted still hold their client on the stack

Likely causes

• •Missing or conditional finally branch: code returns from try/catch without invoking the release/destroy/return method on the checked-out client
• •Client obtained inside one branch and released in another, so an early return, thrown error, or canceled promise skips the release path
• •Streaming response not drained or destroyed: a stream that is paused, errored, or never consumed leaks its socket because the pool expects the connection to be returned on stream end
• •Request aborted via timeout, signal, or AbortController but the abort handler does not call the pool's release/destroy equivalent, so the slot stays occupied
• •Nested or wildcard middleware performs request side-effects and never propagates control back to the original handler that owns the checkout
• •Reused client reference: handing the same pool client to two callers means one caller's release destroys the other's checkout, leaving an orphan reference in the pool's internal list

First ten minutes

1. 01Capture a baseline: in the process or service showing the symptom, snapshot the pool's checked-out count, queue length, and free count once per second for two minutes to confirm they trend up or never return to zero
2. 02Reproduce with isolation: send a single deterministic request that mirrors the failing path, attach a tracer or counter to the checkout and release calls, and confirm the release counter is zero or missing after the response
3. 03Diff request and error paths: list the endpoints or code branches that throw most often in your logs, then compare each branch for the presence of an explicit release call before the function returns
4. 04Inspect streams and aborts: for any endpoint that returns a stream, confirm there is a destroy/close path on error, abort, and client disconnect; note which paths lack one

Evidence to collect

• •Pool state samples over time: free, acquired, queued, and pending destruction counts; correlate with request rate to show non-zero acquired during idle periods
• •Stack traces from the checkout event tagged with a checkout id, taken at both the leak site and a later acquisition, to show the same stack persists across acquisitions
• •Code-path inventory of every site that obtains a client, annotated with whether a release call exists in the success, error, and abort branches
• •Response lifecycle evidence: confirm whether streamed responses are consumed, aborted, or left dangling at the end of each request handler
• •Error and timeout logs for the affected endpoint, since error paths are the most common place to skip a release

Where to look

• •The HTTP client factory and any wrapper that calls a getClient, acquire, or checkout method, including custom interceptors that hand out pool clients
• •Request handlers that return a stream, pipe it, or forward it to a downstream caller, where consume/destroy on error/abort is required to return the socket
• •Error, timeout, and cancellation branches, including try/catch/finally and Promise .then/.catch chains, where a throw before release leaves the client checked out
• •Middleware, retries, and circuit breakers that wrap the client and may swallow or rethrow without delegating release back to the owner
• •Lifecycle hooks tied to abort signals, request timeouts, and server-sent disconnects that must trigger release even when the response handler has not run

Diagnostic steps

1. 01Confirm the leak: plot the pool's acquired count versus time during a quiet period; a non-zero acquired count with zero inflight requests is direct evidence of a leak rather than load
2. 02Take a stack sample at the moment of checkout, tagged with a unique id, and verify that for a leaked checkout the same id is still present in the pool's in-use list minutes later
3. 03Search the codebase for the client acquisition call and rank all call sites by whether they contain a release call in all three branches: success, throw, and abort/timeout
4. 04For stream-returning endpoints, attach a listener on the stream's end, error, and close events to verify that the release runs in each case; missing event handlers on error is a strong leak signal
5. 05Reproduce under controlled load: replay recorded failing requests through a single process and watch whether the pool's acquired count climbs until the queue grows; this isolates the leak from other variability
6. 06Compare leak rate across code versions by running the same replay against a known-good build to verify whether a recent change introduced the missing release

Common mistakes

• •Concluding the pool is too small: raising the pool size masks the leak by spreading it across more sockets rather than fixing it; verified leaks should be capped, not enlarged
• •Adding a release call only in the success branch while leaving throw and abort paths unchanged; this shifts the leak to error traffic and makes the symptom look intermittent
• •Relying on process exit or garbage collection to return clients; pools track live checkouts independently of the runtime, so leaked references persist until restart
• •Wrapping checkout in a higher-order function that catches its own errors and forgets to release before rethrowing, hiding the leak behind friendly error handling
• •Reading the response body to completion but skipping release on the early-exit branch that fires when the body is not needed for the response

Safe fixes

• •Place the release call in a finally block immediately around the smallest scope that owns the checkout, so success, throw, and early return all return the client to the pool
• •For stream-returning handlers, register handlers for stream error and request abort that call the same release function, and ensure the release runs at most once via an idempotent flag
• •Introduce a single pool wrapper that exposes checkout and release as a paired operation and forbid direct access to the underlying factory, so every caller is forced through one release path
• •Tag each checkout with a short, unique id at acquisition time and log it so any in-use list entry without a matching release can be traced back to its request
• •Add a per-request assertion in development that the checkout id present at request start is no longer in the pool's in-use list at request end; failing this assertion indicates a missing release on that specific path

Prove the fix

1. 01Replay the failing request sequence against the fixed build and verify that the pool's acquired count returns to its pre-request baseline within one second of each request's logical end
2. 02Run a soak test that drives the same error-throwing and abort-triggering traffic that previously leaked, and verify the acquired count remains stable over a 30-minute window with no upward drift
3. 03Inspect the checkout-id log and confirm that for every id appearing at request start there is exactly one matching release event at request end, with no orphans after a five-minute idle period
4. 04Confirm the original failure signature is gone under replay: timeouts from "waiting for connection from pool" or pool-closed errors no longer appear in the application's error log during the soak

Prevention and next steps

• •Establish a code-review checklist that rejects any new client acquisition without a paired release in finally, including an explicit error and abort branch
• •Add a periodic background self-check that asserts the pool's in-use list is empty during a known-quiet window and alerts if it is not
• •Centralize client acquisition behind one module so new endpoints cannot call the factory directly, reducing the surface area for missing release calls
• •Cover error and abort paths with tests that simulate thrown requests, aborted responses, and unconsumed streams, and assert the pool returns to empty in each case

Safe commands and checks

ss -tanp state established | awk '{print $5}' | sort | uniq -c | sort -nr | head
ss -tan state established | wc -l
ss -tan state time-wait | wc -l
ss -tan state close-wait | wc -l
ps -o pid,etime,cmd -p <pid>
grep -nE 'acquire|checkout|getClient|release|destroy' <path-to-client-wrapper>.js
grep -rnE 'try\\s*\\{|finally\\s*\\{|catch\\s*\\(' <path-to-http-handlers>
node --inspect=<port> <entry-script>.js