Connection-pool exhaustion: find the borrower that never returned

The symptoms

• •Application logs show `PoolExhausted`, `timeout exceeded when trying to connect`, or `remaining connection slots are reserved` errors at the pool layer rather than the database.
• •Queue wait time metric rises while `idle` connection count collapses to zero and `waiting` client count grows monotonically until the request timeout fires.
• •Steady-state throughput drops even though database CPU, IO, and lock-wait time are low, indicating the bottleneck is at the pool boundary, not the database engine.
• •After a burst, pool size and `idle` never recover to baseline even when incoming request rate has returned to normal levels.
• •Specific endpoints or background jobs hang mid-transaction while health checks and unrelated routes still succeed, pointing to a code path rather than a global outage.

Likely causes

• •A request handler acquires a client and forgets to release it on an exception path, leaking one connection per failed request until the pool is empty.
• •Long-running queries or explicit `SET LOCAL` locks hold a client far longer than expected, so under load the pool drains faster than it refills.
• •Pool sizing is smaller than required concurrency, so a legitimate traffic spike produces queue waits that exceed the application's request timeout.
• •Nested checkout, where code inside one transaction calls another function that also checks out a client from the same pool, deadlocks on a single remaining connection.
• •Network or TCP keepalive failure between application and database causes the client to be marked busy indefinitely until an explicit timeout closes it.

First ten minutes

1. 01Capture the exact pool error string and the time window from the application log; confirm it comes from the pool client rather than the underlying database driver or engine.
2. 02Snapshot pool metrics: `total`, `idle`, `waiting`, and average/maximum queue wait, to determine whether the pool is saturated or genuinely drained.
3. 03Compare the current `total` count against the configured maximum; if `total` equals the maximum and `idle` is zero, the pool is full of checked-out clients.
4. 04Identify the most recent error stack traces that originated near a `connect`, `query`, or `transaction` call so you can correlate them with the metric spike.
5. 05Check whether a deploy, config change, or scheduled job started shortly before the symptom, since pool exhaustion often follows a behavioural change.
6. 06Record a list of every endpoint, worker, or cron task active during the window so you can map borrowers to code paths in the next phase.

Evidence to collect

• •Pool metric time series: `total`, `idle`, `waiting`, `max`, plus pool acquire and query duration percentiles, from the application or APM.
• •Application stack traces showing pool acquire sites, especially those that appear immediately before `PoolExhausted` or timeout log lines.
• •Database-side view of active sessions (for example `pg_stat_activity` for Postgres) including state, duration, and blocking relationships.
• •Recent deploy diffs, config changes, and scheduled-job run history within the same window as the first symptom.
• •Process or thread state dump of the application, capturing which thread currently holds a pooled client and where it is parked.

Where to look

• •The application-to-pool boundary: every `pool.connect`, `pool.acquire`, or equivalent call site, and the matching release path inside the same scope.
• •The pool configuration object or file, where `max`, `min`, `idleTimeoutMillis`, `connectionTimeoutMillis`, and `allowExitOnIdle` are defined.
• •The transaction wrapper or middleware that opens a client, because leaks almost always live in its `try/catch/finally` structure.
• •The database connection view, where long-lived idle-in-transaction or active sessions reveal which application requests still own a client.
• •Background workers and cron schedulers, which often run outside the main request lifecycle and bypass the release discipline used by HTTP handlers.

Diagnostic steps

1. 01Plot `waiting` clients against `idle` clients over time; a curve where `waiting` rises while `idle` is pinned at zero indicates a leak rather than steady saturation.
2. 02Correlate each borrow event with its release event by request ID; any borrow that lacks a matching release is a candidate leak path.
3. 03Cross-reference database-side active sessions with application request IDs to see whether a "stuck" client is genuinely executing or blocked on a network read.
4. 04Reproduce the symptom in a staging environment with the same pool `max` and request mix; if exhaustion does not appear, suspect a configuration regression.
5. 05Bisect by disabling one suspect code path at a time (feature flag or route disablement) and observe whether `total` falls back toward `max - 1`.
6. 06Inspect transaction boundaries for nested checkouts by searching for `pool.connect` calls inside functions that already receive a client as a parameter.
7. 07Verify that every code path that calls `query` directly on the pool uses the implicit checkout and that none rely on a held client from a previous scope.

Common mistakes

• •Increasing `max` without finding the leak, which delays detection and can push the database past its connection ceiling.
• •Adding a generic `catch` that swallows the pool error and retries, masking the underlying borrower that never returned.
• •Restarting the application as the long-term fix, which clears the pool but leaves the leak in code to recur on the next deploy.
• •Assuming the database is slow when pool metrics show the wait is at the application layer, not at the engine.
• •Treating `PoolExhausted` as a capacity problem only, when a single misbehaving endpoint can starve every other route of clients.

Safe fixes

• •If you confirm an unhandled exception path leaks a client, wrap the borrow in a `try/finally` (or equivalent) and release the client on every branch before deploying the fix.
• •When a specific endpoint shows abnormally long client-hold time, cap its per-request query timeout and add a circuit breaker so it cannot monopolise the pool.
• •If nested checkout is the cause, refactor the inner function to accept the existing client rather than requesting a new one from the pool.
• •If pool size is genuinely undersized for the workload, raise it only after measuring required concurrency, and confirm the database can accept the new ceiling.
• •For leaked idle-in-transaction sessions, introduce an idle-transaction timeout at the database so stranded clients are forcibly closed after a defined period.

Prove the fix

1. 01Under the same load profile that previously caused exhaustion, `waiting` clients must remain at or near zero and `idle` must recover to baseline within the configured `idleTimeoutMillis` after traffic subsides.
2. 02Pool acquire duration p99 must drop back to its pre-incident baseline, and `PoolExhausted` log lines must not recur during a sustained soak test.
3. 03A targeted unit or integration test that throws inside the previously leaking code path must show the client being released, verified by `pool.idle` remaining at the expected count.
4. 04Database-side `pg_stat_activity` must show no application session older than the defined transaction timeout during a one-hour load window.

Prevention and next steps

• •Adopt a single pool-acquisition helper that pairs borrow and release in one construct, and forbid raw `pool.connect` calls in code review.
• •Add metrics for `waiting` clients and pool acquire duration, with alerts when `waiting` exceeds a small threshold for more than a few seconds.
• •Set explicit `connectionTimeoutMillis` and `idleTimeoutMillis` values rather than relying on driver defaults, and document them alongside `max`.
• •Periodically run load tests that exceed the configured `max` to verify graceful failure and correct release behaviour under contention.
• •Track database-side idle-in-transaction duration and alert when any session exceeds the application's transaction timeout.

Safe commands and checks

pgrep -af <process_name>  # locate the application process IDs to inspect for thread state
ss -tan state established '( dport = :<port> )'  # list established TCP sessions between application and database on the configured port
ps -o pid,stat,etime,command -L -p <pid>  # enumerate threads of the application process and their current state
jstack <pid> 2>/dev/null | grep -A 20 'pool'  # capture JVM thread stacks filtered for pool-related frames, when running on a JVM
node --inspect-brk=0.0.0.0:0 app.js  # enable a debugging port for the Node.js application so a remote inspector can attach without changing runtime behaviour
SELECT pid, state, now() - state_change AS duration, query FROM pg_stat_activity WHERE datname = current_database() ORDER BY duration DESC;  # inspect active database sessions ordered by duration
SELECT pid, now() - xact_start AS xact_duration, state, query FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY xact_duration DESC;  # find idle-in-transaction sessions that may hold a pool client indefinitely