Connection-pool queue starvation: find borrowers blocking unrelated work

The symptoms

• •Application threads waiting on connection acquisition show monotonically growing wait times even though overall database CPU and IO utilization remain moderate, indicating the pool is full but not all borrowers are doing useful work.
• •Queue depth on the pool's wait queue rises and stays elevated while a small number of connections remain checked out far longer than the request mix would predict, suggesting those borrowers are starving the rest.
• •Unrelated request classes degrade together: a slow report or batch task correlates with increased latency on health checks, login endpoints, or short OLTP queries served from the same pool.
• •PostgreSQL pg_stat_activity shows a near-constant number of active sessions equal to the pool size, with the same backend PIDs associated with the long-running statements while newer application requests pile up on the application side.

Likely causes

• •One or more borrowers run statements that exceed the pool's expected transaction lifetime, such as long reports, admin queries, or schema operations that hold a connection for seconds to minutes.
• •Borrowers do not release connections promptly on error paths, leaking checked-out connections back into the pool only when the application restarts or the socket is reaped.
• •A single shared pool serves workloads with very different latency budgets, so fairness degrades: fast requests queue behind one slow borrower and their combined wait time dominates tail latency.
• •Pool configuration assumes uniform checkout time and uses an unbounded or large wait queue, which hides the starvation rather than surfacing it as a fast failure with a clear error.
• •Connection checkout is not bounded by statement-level timeouts, so a borrower that begins a transaction and then waits on an external dependency keeps the slot indefinitely until the external system responds.

First ten minutes

1. 01Confirm the symptom is at the pool layer, not at the database: capture pool wait-time metrics, queue depth, and checkout duration histograms, and compare them to database CPU, IO, and lock-wait indicators to rule out a pure database bottleneck.
2. 02Snapshot PostgreSQL's pg_stat_activity to record which backend PIDs have been active for the longest, the query text they are running, and the application name reported by each session.
3. 03Identify whether the pool size plus any other consumers of database connections equals or exceeds the database's max_connections setting, because exhaustion of the server side produces starvation-like symptoms at the pool.
4. 04Correlate the long-lived backend PIDs with the application side by matching the application_name, client_addr, or a query identifier to the borrowers that checked out those connections, so you know which code paths to inspect.
5. 05Decide whether the symptom is a single hot borrower or a class of borrowers, because the fix differs: one route can be re-routed, a class may require pool subdivision or a dedicated pool.

Evidence to collect

• •Pool metrics: current pool size, active (checked-out) connections, idle connections, wait queue depth, and p50/p95/p99 wait time over the incident window, with timestamps.
• •Per-connection checkout duration histogram and, where available, the borrower identifier or request ID associated with each long checkout.
• •PostgreSQL pg_stat_activity snapshot: pid, application_name, state, query_start, state_change, wait_event_type, wait_event, and the leading portion of the query text for each active backend.
• •PostgreSQL pg_stat_database and pg_stat_user_tables snapshots for the same window, to confirm whether the database itself is a bottleneck or only the pool is.
• •Workload classification: a coarse map of which application routes or job types map to which borrower classes, so you can reason about fairness instead of guessing from query text.

Where to look

• •The application-side connection pool boundary: its metrics endpoint, its internal counters for checkouts, waiters, and checkout duration, and its configuration for max connections, max queue size, and timeouts.
• •The PostgreSQL monitoring statistics views documented in the official monitoring stats documentation, specifically pg_stat_activity and pg_stat_database, which together describe server-side session state and per-database activity.
• •The application logs that record which request, thread, or coroutine holds which connection, often keyed by a request ID or transaction ID that also appears in database logs.
• •The boundary between the pool and PostgreSQL, where pg_stat_activity and the pool's checkout list must agree on the number of live sessions and their identities.

Diagnostic steps

1. 01Measure the fraction of time the pool spends with active connections equal to its maximum and a non-empty wait queue; a sustained value above the majority of the incident window indicates queue starvation rather than transient contention.
2. 02Rank active sessions in pg_stat_activity by query_start duration and compare that ranking against the pool's longest-held checkouts; agreement confirms that the database sees the same long borrowers the pool does.
3. 03Test the alternative hypothesis that the database is overloaded by reading pg_stat_database for tup_returned, tup_fetched, xact_commit, xact_rollback, and blk_read_time plus blk_write_time; saturation in these counters points away from pool starvation.
4. 04Inspect wait_event_type and wait_event in pg_stat_activity for the long-lived sessions to determine whether they are running CPU, waiting on IO, waiting on a lock, or idle in transaction; each points to a different cause.
5. 05Isolate one suspected borrower class at a time: throttle or disable it in a non-production environment and observe whether pool wait times and queue depth return to baseline, which proves causation for that class.

Common mistakes

• •Increasing the pool size in response to starvation without identifying the long-lived borrower: this often makes the problem worse on the database side because more concurrent sessions increase lock and IO contention without reducing per-borrower checkout time.
• •Treating high CPU or high IO on the database as proof that starvation is a database problem, when the actual cause is that a small number of borrowers occupy the pool and prevent fairness checks from running.
• •Reading pg_stat_activity without comparing application_name, client_addr, or query identifiers to the pool, which leaves you unable to link a long database session back to the responsible code path.
• •Assuming queue depth growth means the pool is undersized; an unbounded queue hides starvation by letting wait times grow without bound instead of failing fast with a clear acquisition-timeout error.
• •Restarting the application as the first response, which resets the pool and briefly restores service but does not change the borrower behavior that produced the starvation, so the symptom returns on the next long-running job.

Safe fixes

• •If a single route or job class is identified as the long-lived borrower, route it to a dedicated pool with its own size and timeouts so its checkout time no longer competes with latency-sensitive traffic on the main pool.
• •If borrowers leak connections on error paths, add a connection-level statement timeout or transaction timeout at the pool layer so a stuck checkout is reaped even when the application code fails to release it.
• •If the pool's wait queue is unbounded or very large, set a finite queue size and a borrow timeout so starvation surfaces as a fast, observable error rather than as unbounded latency growth.
• •If multiple workloads share one pool and have different latency budgets, subdivide the pool by workload class only after evidence shows that one class is the dominant long-lived borrower; do not subdivide preemptively without data.
• •Before any of the above, raise a short-term acquisition timeout in the pool so the failure is observable in dashboards and alerts; keep the change reversible and tied to the same metrics used to confirm starvation.

Prove the fix

1. 01Pool wait-time p95 and p99 return to their pre-incident baseline during a window that includes a representative mix of the previously starving workloads, demonstrating that fairness is restored.
2. 02Queue depth remains bounded under the same workload mix, with the configured finite queue size and acquisition timeout producing explicit acquisition-timeout errors rather than silent latency growth when the pool is saturated.
3. 03PostgreSQL pg_stat_activity shows that no single application_name or query class holds a connection for longer than the configured statement or transaction timeout, confirming that long-lived borrowers are now bounded.
4. 04An end-to-end regression check on a representative short request class returns to its pre-incident latency distribution even while the previously long-running class runs concurrently, proving the fix restored concurrency rather than merely reducing load.
5. 05A follow-up comparison of pg_stat_database activity before and after the change shows no unintended increase in commits, rollbacks, or IO that would indicate the fix merely shifted the bottleneck.

Prevention and next steps

• •Define a per-borrower checkout budget, expressed as a maximum checkout duration, and enforce it with a statement or transaction timeout at the pool layer so long borrowers fail fast instead of starving the queue.
• •Separate pools by workload class whenever classes with very different latency budgets share the same database, and size each pool against the database's max_connections budget rather than against an unbounded number of workers.
• •Monitor pool wait-time percentiles, queue depth, and checkout duration alongside PostgreSQL pg_stat_activity and pg_stat_database so starvation is detected at the boundary, not only after user-visible latency degrades.
• •Test long-running and short-running workloads together in pre-production to confirm that the pool configuration yields bounded queue depth and a non-degrading short-request latency distribution before deployment.

Safe commands and checks

psql -h <db_host> -p <port> -U <db_user> -d <db_name> -c "SELECT pid, application_name, client_addr, state, query_start, now() - query_start AS duration, wait_event_type, wait_event, left(query, 200) AS query_preview FROM pg_stat_activity WHERE state IS NOT NULL ORDER BY query_start;"
psql -h <db_host> -p <port> -U <db_user> -d <db_name> -c "SELECT datname, numbackends, xact_commit, xact_rollback, tup_returned, tup_fetched, blk_read_time, blk_write_time FROM pg_stat_database WHERE datname = current_database();"
psql -h <db_host> -p <port> -U <db_user> -d <db_name> -c "SELECT count(*) FILTER (WHERE state = 'active') AS active, count(*) FILTER (WHERE state = 'idle in transaction') AS idle_in_tx, count(*) FILTER (WHERE state = 'idle') AS idle FROM pg_stat_activity;"
psql -h <db_host> -p <port> -U <db_user> -d <db_name> -c "SELECT application_name, count(*) AS sessions, max(now() - query_start) AS oldest FROM pg_stat_activity WHERE state IS NOT NULL GROUP BY application_name ORDER BY oldest DESC;"