API is slow after idle periods: investigate connection warm-up

The symptoms

• •First request after an idle window (typically seconds to tens of minutes) is markedly slower than the median request, often by 2x to 10x or more.
• •Latency spike is concentrated in the very first request; a second request issued within a few seconds returns to normal timing.
• •Spike is reproducible by letting traffic fall to zero (or near zero) and then issuing a single synthetic call.
• •Distribution shows a long thin tail dominated by low-traffic or off-hour windows rather than by peak-hour load.
• •User-facing impact is reported as slow first page loads after deployment, after scale-in events, or after overnight quiet periods.

Likely causes

• •HTTP connection pool shrank or closed during the idle window and the next request must perform TCP handshake, TLS handshake, and HTTP/2 negotiation from scratch.
• •Backend service process or instance was scaled to zero or hibernated and must be cold-started before serving the first request.
• •Downstream dependency (database connection pool, cache, message broker, third-party API) closed its pooled sessions and must re-establish them on the first request.
• •Just-in-time compiler or runtime optimizer warmed code paths out and must re-JIT or re-cache class metadata, bytecode, or query plans on the first request.
• •DNS, service-mesh sidecar, or API gateway attachment state was reaped during inactivity and must be re-attached when traffic resumes.
• •Garbage collector or runtime allocator reset warm heap state (young generation, code cache, tiered caches) during the idle window.

First ten minutes

1. 01Reproduce the symptom in a controlled way: let the endpoint sit idle for a known quiet window (for example, the longest observed gap) and then issue a single timed request; record the latency and the next-request latency for comparison.
2. 02Capture one idle-window request with a request-level trace or APM span and identify which segment of the call (DNS, connect, TLS, server processing, backend call) accounts for the extra time.
3. 03Plot latency by request-order-since-traffic to see whether only request #1 is slow or whether a small burst is required to reach steady state.
4. 04Check whether the slow window correlates with autoscaler scale-in or scale-to-zero events, deploy rollouts, or instance recycling by cross-referencing APM timestamps with platform events.
5. 05Compare a cold-region or cold-tenant request against a warm-region or warm-tenant request under identical load to isolate whether the cost is paid per process, per pool, or per dependency.

Evidence to collect

• •Two consecutive request latencies after a known idle window, with the idle gap duration recorded alongside.
• •Distributed trace spans broken down by DNS, connect, TLS, server processing, and backend dependency segments for the first and second request.
• •Connection pool size and active count at the moment of the first request, taken from the runtime or proxy metrics.
• •Autoscaler, scheduler, or platform events for the affected instance covering the idle window (scale-in, eviction, cold start, deployment).
• •Backend dependency metrics (database, cache, broker) showing session, connection, or pool counts during the idle window and at first-request time.
• •Histogram of latency bucketed by hour-of-day or by requests-since-last-request to confirm the tail is concentrated in idle periods.

Where to look

• •At the client boundary: keep-alive and pooling settings on the HTTP client, including idle timeout, pool size, and connection reuse policy.
• •At the server boundary: the load balancer, reverse proxy, or API gateway in front of the service, including its idle connection reaping and TLS session resumption settings.
• •Inside the application: startup time, classpath or module load, cache warm-up, and JIT/optimizer thresholds triggered by the first request.
• •At the dependency boundary: database driver pool settings, ORM lazy initialization, cache client bootstrap, message broker consumer reconnect, and third-party API client setup.
• •At the platform boundary: autoscaler minimum instance count, scale-to-zero behavior, instance warm-pool settings, and cold-start mitigation features.

Diagnostic steps

1. 01Measure the first-request-after-idle latency versus steady-state latency and compute the ratio; a large ratio with a single-request spike is consistent with warm-up cost rather than a capacity regression.
2. 02Inspect the connect and TLS segments of the trace; if these dominate the extra time, the cost is paid at the network boundary and points at pool or keep-alive configuration.
3. 03Inspect server processing and backend dependency segments; if these dominate, the cost is paid inside the process or at a downstream call and points at runtime warm-up or dependency pool growth.
4. 04Compare two requests issued back-to-back after the same idle window; if only the first is slow, the cost is one-shot (setup) rather than recurring (capacity).
5. 05Correlate the slow first request with platform cold-start events; if a match is observed, the cost is paid at instance provisioning rather than at the application layer.
6. 06Disable or reduce one warm-up dependency at a time (for example, pre-create a database pool) and re-measure; a reduction in first-request latency isolates that dependency as a contributor.

Common mistakes

• •Concluding the service is overloaded because p99 latency spikes, when the spike is actually concentrated in the first request after idle and steady-state p99 is healthy.
• •Adding more replicas or raising CPU limits in response to idle-slow symptoms, which does not change the per-process warm-up cost paid on the first request.
• •Disabling connection pooling entirely as a quick fix, which increases per-request overhead and can shift the latency problem onto every request rather than the first.
• •Treating the symptom as a code regression after a deploy, when the slow first request is caused by a fresh process paying warm-up cost rather than by a new bug.
• •Raising client or proxy idle timeouts to very large values without checking upstream limits, which can cause socket, file descriptor, or backend pool exhaustion.

Safe fixes

• •If connect and TLS dominate, increase HTTP client keep-alive idle timeout and pool size so connections survive the idle window, and enable TLS session resumption or session tickets where supported.
• •If server processing dominates, move expensive one-time initialization (caches, regex compilation, configuration parsing) out of the request path into startup, and verify JIT or optimizer warm-up is complete before serving traffic.
• •If a downstream dependency dominates, configure its client to pre-create a minimum pool size at startup and to keep a small number of sessions warm during idle windows.
• •If platform cold start dominates, raise the autoscaler minimum instance count above zero, or use a warm-pool or provisioned concurrency feature so at least one instance is always initialized.
• •Once a fix is applied, define a proof step that re-runs the idle-then-first-request measurement and confirms the first-request latency is within an acceptable margin of the steady-state latency.

Prove the fix

1. 01Run the original reproduction: an idle window of the same duration, followed by one timed request, then a second timed request. The first-request latency should now be within a defined margin (for example, less than 1.5x) of the second-request latency.
2. 02Repeat the measurement across multiple idle windows of varying length and confirm the first-request latency does not grow with idle gap, indicating the warm-up cost is bounded.
3. 03Re-inspect the trace for the first request after the fix and confirm the segment that previously dominated (connect, TLS, server, or dependency) is no longer the outlier.
4. 04Monitor p99 latency bucketed by hour-of-day or by requests-since-last-request and confirm the idle-window tail has receded toward the steady-state band.
5. 05Document a regression check that fails CI or alerting if the first-request-after-idle ratio exceeds a defined threshold, so the warm-up cost cannot silently return.

Prevention and next steps

• •Keep HTTP client pools and keep-alive idle timeouts aligned with the longest expected idle gap between requests, within upstream and resource limits.
• •Set a non-zero minimum instance count or use warm-pool features so the platform does not cold-start a fresh process on the first request after quiet periods.
• •Move one-time initialization, caching, and connection pool pre-warming out of the request path and into process startup or pre-serve hooks.
• •Add a synthetic warm-up probe or low-rate traffic source during quiet windows to keep pools and JIT state warm without raising user-facing load.
• •Track a first-request-after-idle latency metric and alert if its ratio to steady-state latency drifts upward, since regressions often appear here before they surface in aggregate p99.

Safe commands and checks

ps -o pid,etime,cmd -p <pid> | head -n 5
ss -tan state time-wait | awk 'NR>1 {print $4}' | sort | uniq -c | sort -rn | head -n 10
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c
cat /proc/<pid>/status | grep -E '^(Threads|VmRSS|VmSize|VoluntaryCtxtSwitch|NonVoluntaryCtxtSwitch):'
date -u +%FT%TZ
awk '{print $1}' /var/log/<service>/access.log | sort | uniq -c | tail -n 5
awk '{print $4}' /var/log/<service>/access.log | sort | head -n 20