API client connection-pool checklist

The symptoms

• •Intermittent 'socket hang up' or ECONNRESET errors that correlate with traffic bursts rather than with specific endpoints
• •Requests queue or time out after the first wave of concurrent calls, even though the target host is healthy
• •Pool size metric plateaus at a value below the configured maximum, indicating sockets are acquired but never released
• •Open file descriptor count on the client process grows during load and only drops after process restart
• •Idle sockets visible in pool stats that never get reused, suggesting keep-alive or timeout configuration is off

Likely causes

• •Missing or incorrect keep-alive header handling, so sockets close after every request instead of returning to the pool
• •Pool maximum connections set lower than peak concurrency, causing legitimate requests to queue or fail
• •Response streams or response objects not fully consumed on error paths, leaking sockets out of the pool
• •Per-host connection caps unintentionally applied to a shared agent that fronts multiple upstream hosts
• •Unbounded retry or backoff loops holding sockets while waiting for the upstream to recover
• •DNS or TLS errors that abort the socket mid-handshake and are not treated as pool-releasing failures

First ten minutes

1. 01Confirm the failure mode is pool-related by checking whether the error rate rises with concurrency, not with a specific URL path
2. 02Capture the exact error class and code from the client error object and record it alongside the configured pool max and per-host max
3. 03Read the connection event counters exposed by the client: acquired, released, free, queued, and pending timeouts
4. 04Inspect the process open file descriptor count before and after a short load window to see if sockets accumulate
5. 05Compare configured timeouts against upstream median latency to decide whether sockets are being held by slow responses rather than leaked
6. 06Decide whether to attribute the symptom to configuration, calling code, or upstream behavior before changing any value

Evidence to collect

• •Error class, error code, and message text from the client error object for at least ten recent failures
• •Pool stats snapshot showing acquired, free, queued, and pending socket counts at the moment of failure
• •Configured values for max connections, max connections per host, keep-alive timeout, and socket timeout
• •Open file descriptor count sampled at idle and at peak load on the client process
• •Distribution of request durations so you can separate slow upstream from pool starvation

Where to look

• •The HTTP client agent configuration object in your bootstrap file, where maxSockets, maxFreeSockets, and timeouts are set
• •Error handlers and catch blocks around outbound calls, where uncaught rejections can prevent socket release
• •Request wrappers that add retries, circuit breakers, or auth refresh, since they often wrap the underlying client call
• •Process and container resource metrics, where file descriptor counts and event loop lag are visible
• •Library telemetry endpoints or debug logs that expose acquire, release, and timeout events for the pool

Diagnostic steps

1. 01Reproduce the symptom under controlled concurrency by issuing a fixed number of parallel requests to a known-good endpoint and watching the error rate rise above zero
2. 02Capture pool acquire and release counts during the run; if releases lag acquires by the full request count, sockets are not returning to the pool
3. 03Toggle keep-alive off and on in a test environment and compare the number of sockets opened during the same workload
4. 04Lower concurrency below the configured max and confirm errors disappear, which separates pool exhaustion from upstream brownouts
5. 05Force an error path in a test and verify the response stream is fully consumed or the connection is explicitly destroyed on failure
6. 06Trace a single failing request end to end to see whether a retry wrapper or auth refresh is holding the socket during the wait

Common mistakes

• •Raising the pool maximum without first proving the sockets are actually leaked rather than simply slow to return
• •Setting keep-alive timeout to zero, which silently disables reuse and forces a new socket for every request
• •Sharing a single global agent across services with different latency profiles, so one slow upstream stalls every consumer
• •Swallowing errors in catch blocks without releasing the response, which leaks the socket for the lifetime of the agent
• •Reading pool stats only at process start, so a one-time spike is mistaken for steady-state exhaustion

Safe fixes

• •If releases lag acquires, fix the call sites first: consume the response on every code path and destroy the request on error, then re-measure before changing limits
• •If pool max is below measured peak concurrency and releases are timely, raise the limit to a value justified by the load test, not by guesswork
• •If keep-alive is the cause, set an explicit keep-alive timeout derived from the upstream idle close policy rather than disabling reuse outright
• •If a retry wrapper holds sockets, bound the retry budget and add jitter so a stalled upstream cannot pin the entire pool
• •If per-host caps throttle a shared agent, partition the agent per upstream host so one busy target cannot starve the others

Prove the fix

1. 01Acquire minus release returns to zero within the keep-alive window after the workload ends, with no residual queued count
2. 02Open file descriptor count stabilizes under repeated load runs instead of climbing each iteration
3. 03Error rate under the original failing concurrency drops to the baseline observed below the pool maximum
4. 04Pool stats show sockets moving between acquired and free states, proving reuse rather than one-shot allocation
5. 05A targeted fault-injection test that aborts responses on the error path no longer increases the in-use socket count

Prevention and next steps

• •Expose pool acquire, release, free, and queued counters as metrics so exhaustion is visible before it becomes an outage
• •Add a load test in CI that asserts acquires equal releases at the end of the run, catching leaks before deploy
• •Document the chosen pool maximums alongside the latency profile that justified them so future changes do not regress them silently
• •Lint for unhandled response streams in outbound call sites and fail the build when a response can escape without being consumed or destroyed

Safe commands and checks

node -e 'const a=require("http").globalAgent; console.log({maxSockets:a.maxSockets, maxFreeSockets:a.maxFreeSockets, keepAlive:a.keepAlive})'
node -e 'const a=require("http").globalAgent; setInterval(()=>{const s=a.sockets; const f=a.freeSockets; let inUse=0,free=0; for(const k of Object.keys(s)) inUse+=s[k].length; for(const k of Object.keys(f)) free+=f[k].length; console.log({inUse,free,queued:a.requests&&Object.values(a.requests||{}).reduce((n,r)=>n+(r&&r.length||0),0)})},1000)'
node -e 'const c=require("http"); const req=c.request({host:"<upstream-host>",port:<port>,path:"/health",method:"GET",agent:false},(res)=>{res.on("data",()=>{}); res.on("end",()=>console.log(res.statusCode))}); req.on("error",(e)=>console.error(e.code,e.message)); req.end()'
node -e 'console.log(process.report.getReport().libuv.map(u=>u))' 2>/dev/null | head -n 40
node -e 'process.on("warning",(w)=>console.warn(w.name,w.message))'  # run while reproducing to surface socket-leak warnings
node --trace-warnings -e 'setInterval(()=>{const a=require("http").globalAgent; for(const k of Object.keys(a.sockets||{})) if(a.sockets[k].length>50){console.error("possible leak",k,a.sockets[k].length)}},2000)'
node -e 'const http=require("http"); const agent=new http.Agent({keepAlive:true,maxSockets:25,maxFreeSockets:10,timeout:30000,freeSocketTimeout:15000}); console.log(Object.getOwnPropertyNames(agent.__proto__).sort())'