PostgreSQL pool queue grows while CPU is low: identify waiting clients

The symptoms

• •Application latency climbs and request timeouts appear, but `pg_stat_activity` shows most sessions in an idle or low-CPU state and server-wide CPU load stays well below saturation.
• •Connection pool metrics (HikariCP, PgBouncer, RDS Proxy, application-side pools) report a non-zero waiting-thread count and rising wait time per acquisition.
• •PostgreSQL `pg_stat_activity.wait_event` is non-null for many sessions, with values such as `Lock`, `LWLock`, `ClientWait`, or `IO` rather than CPU-bound `CPU*` events.
• •Throughput plateaus even though additional pool capacity appears available, suggesting logical rather than compute limits.

Likely causes

• •Pool exhaustion: the maximum pool size is smaller than the number of concurrent request threads, so threads queue waiting for a free connection.
• •Lock contention: long-running transactions or row updates hold row, page, or transaction locks that block other writers and readers, while CPU stays low.
• •Idle-in-transaction sessions: clients opened a transaction and never closed it, holding snapshots and locks while doing no CPU work.
• •Synchronous single-row hot-spot updates: many sessions serialize on the same tuple, producing high `Lock` wait time with little CPU use.
• •Slow consumers: the application reads large results slowly, keeping connections busy in `ClientRead` / `ClientWrite` states and starving other waiters.

First ten minutes

1. 01Snapshot waiting clients with `pg_stat_activity`: capture `state`, `wait_event_type`, `wait_event`, `query_start`, `state_change`, and `pid` for every non-idle backend.
2. 02Snapshot pool-side waiters: export the current pool's waiting-thread count and mean/max wait time from the application or proxy.
3. 03Classify each non-idle backend's wait_event into one of: lock, I/O, client, extension, IPC, timeout, or CPU to decide the dominant wait class.
4. 04Identify sessions in `idle in transaction` or `idle in transaction (aborted)`; these are a leading cause of "queue grows but CPU is low."
5. 05Record the statement text and lock target (relation, tuple) for the oldest waiters so the contention source can be mapped back to the application path.
6. 06Compare server CPU and runqueue against pool wait count: if CPU is low but waits are high, the bottleneck is not compute.

Evidence to collect

• •`pg_stat_activity` rows with non-null `wait_event` and `wait_event_type`, including their `state`, `query`, and age.
• •Pool metrics: total connections, active connections, idle connections, waiting threads, and p95/p99 connection-acquisition latency.
• •`pg_locks` joined with `pg_stat_activity` to show which sessions hold which locks and which sessions are blocked.
• •`pg_stat_database` counters such as `xact_commit`, `xact_rollback`, `conflicts`, and `deadlocks` for a baseline of contention rate.
• •Application-side slow-query log or trace IDs corresponding to the oldest waiting statements.

Where to look

• •Server boundary: `pg_stat_activity`, `pg_locks`, and `pg_stat_database` in the target database; the `stats` views described in the PostgreSQL monitoring stats documentation.
• •Application boundary: the JDBC/ODBC/ORM pool configuration and its metrics endpoint, plus request traces that time the database call separately from CPU time.
• •Proxy boundary: PgBouncer's `SHOW POOLS`, `SHOW CLIENTS`, and `SHOW SERVERS` if a transaction-pooling proxy sits in front of PostgreSQL.
• •Workload boundary: the longest-running transactions and the queries touching the most contended relations or tuples.

Diagnostic steps

1. 01Run a snapshot query against `pg_stat_activity` filtered to `state <> 'idle'` and group by `wait_event_type` to confirm the dominant wait class; treat a non-trivial `Lock` or `LWLock` share as evidence of contention rather than CPU limits.
2. 02Join `pg_locks` to `pg_stat_activity` on `pid` and use `pg_blocking_pids(pid)` to identify the head of each lock chain, then inspect the blocking session's `query` and `xact_start` to localize the slow transaction.
3. 03Compare pool max size, active count, and waiting count; if active equals max and waiting is non-zero, the queue is connection exhaustion, not lock contention.
4. 04Check `pg_stat_activity` for sessions in `idle in transaction` older than the application's expected transaction lifetime; these are usually the lock holders.
5. 05Inspect `wait_event` values of `ClientRead`, `ClientWrite`, or `ClientWait` to detect slow consumers that are starving the pool even though CPU is free.
6. 06Cross-reference the oldest waiting statement's `query` with the application's slow-query log to map the symptom back to a code path.

Common mistakes

• •Assuming high latency means the database CPU is saturated; on PostgreSQL, lock and client waits do not register as CPU load and require separate evidence.
• •Increasing `max_connections` or pool size without checking `wait_event` first, which can amplify lock contention and idle-in-transaction retention.
• •Reading only the application pool's "active" count and missing the "waiting" count, which is the actual signal of queue growth.
• •Treating `idle in transaction` sessions as harmless because they consume no CPU, even though they hold locks and snapshots that block others.
• •Conflating PgBouncer's `sv_active` with database load; the proxy shows connection activity, not server compute, and can hide slow queries.

Safe fixes

• •If `wait_event_type = 'Lock'` dominates and blockers are in `idle in transaction`, set a server-side `idle_in_transaction_session_timeout` and a statement timeout to release stuck transactions; verify by watching the `idle in transaction` count drop.
• •If pool `active == max_size` with non-zero `waiting`, first reduce per-request work or transaction length, then raise pool size only after measuring lock contention; verify by observing waiting count fall while p95 latency improves.
• •If `wait_event` is `ClientRead` / `ClientWrite`, fix the slow consumer (fetch size, streaming, batch size) before changing pool limits; verify by watching wait events shift off `Client*` and queue length shrink.
• •If lock chains converge on a single relation or tuple, batch the hot-spot updates or use `SELECT ... FOR UPDATE SKIP LOCKED` to break serialization; verify by checking that `pg_locks` blocked counts drop while throughput recovers.
• •If pool exhaustion is the dominant signal, cap concurrent request threads in the application to match pool capacity so waits do not pile up; verify by watching waiting count approach zero under the same workload.

Prove the fix

1. 01The pool's waiting-thread count returns to near zero during the same workload that previously queued requests.
2. 02`pg_stat_activity` shows most backends with null `wait_event` or `wait_event_type = 'CPU'` while CPU remains low and latency improves.
3. 03`pg_locks` shows no growing blocked-pid chains; `pg_blocking_pids` results for the top statements return empty or short chains.
4. 04The count of sessions in `idle in transaction` remains within the application's expected steady-state band, not trending upward.
5. 05Application p95 latency for the affected code path improves while server CPU usage does not increase, demonstrating the queue, not compute, was the bottleneck.

Prevention and next steps

• •Set `idle_in_transaction_session_timeout` and a per-statement timeout at the server or pool layer to bound how long a backend can hold locks without making progress.
• •Size the application pool to match the workload's concurrency target and cap request threads accordingly; do not let request threads exceed pool capacity.
• •Adopt a "commit early" discipline: close transactions as soon as the write batch is done, especially around ORM boundaries that leave connections open between calls.
• •Monitor pool wait time, not just CPU, and alert on sustained non-zero waiting-thread counts; review `pg_stat_activity` snapshots as part of the regular on-call checklist.
• •Use `FOR UPDATE SKIP LOCKED` or queue tables for contended hot rows so workers do not serialize on a single tuple.

Safe commands and checks

SELECT pid, usename, state, wait_event_type, wait_event, query_start, state_change, LEFT(query, 200) AS query_snippet FROM pg_stat_activity WHERE state <> 'idle';
SELECT state, wait_event_type, wait_event, count(*) FROM pg_stat_activity WHERE state <> 'idle' GROUP BY 1,2,3 ORDER BY count(*) DESC;
SELECT blocked_locks.pid AS blocked_pid, blocking_locks.pid AS blocking_pid, blocked_activity.query AS blocked_query, blocking_activity.query AS blocking_query FROM pg_locks blocked_locks JOIN pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid JOIN pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype AND blocking_locks.pid <> blocked_locks.pid AND (blocking_locks.granted AND NOT blocked_locks.granted) JOIN pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid WHERE NOT blocked_locks.granted;
SELECT pid, pg_blocking_pids(pid) AS blocking_pids, wait_event, LEFT(query, 200) AS query_snippet FROM pg_stat_activity WHERE wait_event IS NOT NULL;
SELECT datname, xact_commit, xact_rollback, conflicts, deadlocks FROM pg_stat_database WHERE datname = current_database();
SELECT count(*) FILTER (WHERE state = 'idle in transaction') AS idle_in_tx, count(*) FILTER (WHERE wait_event IS NOT NULL) AS waiting, count(*) AS total FROM pg_stat_activity;