CPU is low but requests time out: look beyond compute saturation

The symptoms

• •Application or service CPU utilization stays below a moderate fraction of cores while p95/p99 latency or upstream timeouts spike.
• •Error rate rises with 504 Gateway Timeout, 499 client closed, or read timeout messages even though the load average or container CPU throttle counters remain low.
• •Threads, goroutines, or async tasks appear parked on I/O, mutex, or condition variables instead of running on CPU.
• •Connection or worker pool wait queues grow while CPU usage is flat, suggesting the worker pool is the actual bottleneck.
• •Downstream call deadlines expire before the local service returns, with no local CPU saturation to explain the delay.

Likely causes

• •Thread, request, or async worker pool is too small relative to incoming concurrency, causing requests to queue behind a tight pool rather than consuming CPU.
• •External dependency such as database, cache, object storage, or third-party API is slow, blocking many in-flight requests on a single shared I/O boundary.
• •Lock contention or single-writer data structure serializes work so threads spin or sleep instead of advancing on CPU.
• •Outbound connection limits, file descriptor exhaustion, or socket send/receive buffer saturation stall network I/O even though the process is otherwise idle on CPU.
• •Client-side or upstream timeout is shorter than realistic end-to-end latency, so requests are cancelled before any compute has a chance to fail.
• •Garbage collection pauses, runtime scheduler stalls, or virtual machine safe-points suspend execution while CPU usage looks low at the sampling interval.

First ten minutes

1. 01Capture the time window of the symptom and note CPU, memory, and request rate metrics side by side to confirm the mismatch: low CPU with elevated latency or error rate.
2. 02List active waits using the runtime profiler or kernel sampler: off-CPU stacks, thread states, or runtime trace events, and group them by wait reason.
3. 03Measure current versus configured limits for thread, worker, and connection pools, plus the depth of any wait queue each pool exposes.
4. 04Profile the slowest external dependency with a focused span: database, cache, queue, or third-party API, recording client-perceived latency versus local processing latency.
5. 05Compare the configured upstream timeout against observed end-to-end latency under the same load to decide whether the timeout is the limiting factor.
6. 06Decide which boundary dominates the wait budget before considering any code or configuration change.

Evidence to collect

• •CPU utilization, throttle counters, and load average for the affected process and host during the incident window.
• •Off-CPU or wait-state profile samples tagged by wait reason such as I/O, lock, park, or sleep, with counts and durations.
• •Pool saturation metrics: active versus maximum threads, worker tasks, database connections, and outbound HTTP connections, including queue wait time.
• •Distributed trace spans for a slow request showing time spent in the application code versus in dependencies and serialization.
• •Network and socket statistics: file descriptor usage, TIME_WAIT count, send and receive buffer occupancy, retransmits, and connect attempts.
• •Error logs filtered for timeout, deadline exceeded, pool exhausted, and connection refused events with timestamps and trace identifiers.

Where to look

• •Runtime and language boundary: thread or goroutine scheduler, garbage collector, and any blocking I/O or synchronization primitive inside the process.
• •Concurrency boundary: configured thread pool, async worker pool, connection pool, and bounded queue feeding the request handler.
• •Network boundary: kernel socket buffers, ephemeral port range, NAT or load balancer connection limits, and TLS handshake time.
• •Dependency boundary: database, cache, message queue, object storage, and any synchronous third-party API call within the request path.
• •Client and edge boundary: load balancer, service mesh, and upstream client timeout and retry configuration relative to service-side latency.

Diagnostic steps

1. 01Confirm the mismatch hypothesis by plotting CPU usage against request latency and error rate for the same window; a flat CPU with rising latency supports the non-CPU bound theory.
2. 02Aggregate off-CPU stacks by wait reason and rank them by total time to identify the dominant wait class such as I/O, lock, or pool queue.
3. 03For each pool, compare active count, configured maximum, and observed queue depth to detect saturation without CPU pressure.
4. 04For each external dependency, separate client-side wait time from server processing time using spans and dependency metrics, then identify the slowest link.
5. 05Inspect the shortest configured timeout in the request path and compare it to the p99 end-to-end latency under the same load to test the deadline hypothesis.
6. 06Check kernel and socket statistics for evidence of connection, descriptor, or buffer exhaustion that would stall I/O independently of CPU.
7. 07Synthesize a ranked list of candidate bottlenecks ordered by observed wait time, then design a verification experiment for the top candidate before any change.

Common mistakes

• •Increasing CPU limits, replicas, or instance size when the bottleneck is a bounded pool, a slow dependency, or a too-short timeout, which raises cost without reducing wait time.
• •Treating low CPU as evidence of headroom and adding more concurrency, which deepens queueing and can amplify the timeout storm.
• •Reading thread dumps or stack traces only on-CPU, missing parked threads, blocked I/O, or runtime-specific sleep states that dominate wait time.
• •Assuming the slowest span in a distributed trace is the cause without checking whether the local handler is queuing behind a tight worker pool before it ever runs.
• •Raising client or upstream timeouts without first confirming the upstream can actually complete, which converts timeouts into long, hung requests that exhaust resources.
• •Ignoring safe-points, stop-the-world pauses, or runtime scheduler stalls because aggregate CPU usage at coarse sampling intervals looks low.

Safe fixes

• •If pool wait time dominates, raise the pool size or switch to a bounded async model only after measuring diminishing returns and confirming the downstream can absorb the extra load.
• •If a specific dependency dominates client-perceived latency, add a circuit breaker, cache, or read replica with explicit timeout budgets, and verify with span-level metrics before and after.
• •If lock contention is the top wait reason, reduce critical section size, shard the protected structure, or move work off the hot path, and re-profile to confirm wait time drops.
• •If the configured timeout is shorter than realistic latency, extend it only after increasing local resilience, and verify by observing fewer premature cancellations at the same load.
• •If file descriptors or ephemeral ports are exhausted, raise the limit at the documented kernel or container boundary, monitor for regressions, and confirm via socket statistics.
• •Prefer configuration or topology changes with measurable rollback criteria over speculative code rewrites when the evidence points to a resource ceiling rather than a logic defect.

Prove the fix

1. 01Repeat the off-CPU profile at the same load and confirm the previously dominant wait reason drops in rank and total time while CPU usage stays within budget.
2. 02Replay the same traffic pattern and observe that p95 and p99 end-to-end latency decrease and the timeout error rate returns to the pre-incident baseline.
3. 03Verify that the chosen pool stays below its configured maximum under peak load and that its queue depth does not grow without bound during a sustained test.
4. 04Confirm that no new error class appears, such as connection reset or pool exhausted, after the change, by checking the same error log filter used during diagnosis.
5. 05Document the bottleneck boundary, the change made, and the regression check so that the next on-call engineer can repeat the verification in under ten minutes.

Prevention and next steps

• •Track wait-class metrics alongside CPU metrics so that low CPU does not silently mask I/O, lock, or pool saturation in dashboards and alerts.
• •Set timeouts on every outbound call with budgets shorter than the upstream client timeout, and alert when the budget is regularly exceeded.
• •Size thread, worker, and connection pools with explicit saturation alerts tied to queue depth, not only to utilization.
• •Periodically replay synthetic load to surface dependency latency drift before it crosses user-visible timeout thresholds.
• •Review client and upstream timeout settings together so a tight client deadline cannot cancel requests the service could still complete.

Safe commands and checks

ps -o pid,pcpu,pmem,etime,cmd -p <pid>
top -b -n 1 -p <pid>
ss -s
ss -tan state time-wait | wc -l
cat /proc/<pid>/status | grep -E 'Threads|VmRSS|FDSize'
ls /proc/<pid>/fd | wc -l
cat /proc/sys/net/ipv4/ip_local_port_range
ulimit -n