How to verify graceful shutdown under in-flight requests

The symptoms

• •Active HTTP connections are reset with ECONNRESET or RST during a rolling restart, surfacing 5xx spikes correlated to deploy events.
• •Long-running requests (uploads, streaming responses, websocket upgrades) are aborted mid-flight rather than allowed to complete within a shutdown window.
• •Process exits before the configured shutdown grace window elapses, indicating the signal handler is missing or the event loop is being torn down prematurely.
• •New requests continue to be accepted after SIGTERM, contradicting the "drain" phase of a graceful shutdown policy.
• •Deployment system reports the pod as "ready" again before in-flight work finishes, hiding unfinished requests behind a healthy probe.

Likely causes

• •The process does not register a handler for SIGTERM (or platform equivalent), so the runtime default is immediate termination once the orchestrator sends the termination signal.
• •A graceful shutdown hook exists but does not call server.close() or equivalent, so existing keep-alive sockets are not asked to finish their current request.
• •Shutdown grace period is shorter than realistic request completion time, so requests longer than the window are forcibly killed by the orchestrator's terminationGracePeriodSeconds.
• •Readiness probe is not gated on in-flight drain, so the load balancer shifts traffic to a replacement instance before draining completes.
• •Keep-alive sockets from upstream proxies keep the listener busy, preventing server.close() callback from firing within the budget.
• •Worker threads or background queues ignore the shutdown signal and continue to dispatch work to the request path after the listener has been told to stop.

First ten minutes

1. 01Confirm the failure surface: identify whether 5xx errors, connection resets, or premature exit codes correlate with a deploy or restart event in the orchestration timeline.
2. 02Locate the process signal contract: read the deployment manifest entry for terminationGracePeriodSeconds, preStop hook, and the container's PID 1 command, and compare it to the shutdown handler documented in the application.
3. 03Capture the last shutdown log lines: identify the timestamp of SIGTERM reception, the shutdown hook entry, the listener close invocation, and the process exit code, in that order.
4. 04Quantify in-flight work at the moment of shutdown: record the number of active requests, their elapsed time, and the longest-pending connection, so the policy window can be evaluated against real load.
5. 05Decide which layer failed first: signal handling, listener drain, hook ordering, or orchestrator grace period, before touching configuration.

Evidence to collect

• •Orchestration event log entries pairing the termination signal timestamp with the container's reported exit code and reason.
• •Application shutdown sequence logs marking SIGTERM reception, server.close() invocation, the "drained" callback, and the subsequent process.exit() call, with their deltas.
• •Active request count and request elapsed-time histogram captured at the SIGTERM timestamp, plus the count of requests that did not produce a 2xx response.
• •Listener state showing new connection attempts accepted vs rejected during the drain window, and the time from SIGTERM to "no active sockets".
• •Reverse proxy or load balancer access log slice spanning SIGTERM ± grace period, showing client-side connection resets and response status codes.
• •Process tree and PID 1 identification proving whether a signal trap is in place at the top-level process or only inside a child.

Where to look

• •The deployment manifest boundary: terminationGracePeriodSeconds, preStop lifecycle hook, and the container command that becomes PID 1.
• •The process signal boundary: the PID 1 process and its signal handlers, and any init wrapper that may swallow SIGTERM before the application sees it.
• •The HTTP listener boundary: server.close() / equivalent drain callback, keep-alive socket state, and the readiness probe endpoint.
• •The request handling boundary: in-flight request registry or middleware that tracks active handlers and exposes the drain completion condition.
• •The upstream boundary: reverse proxy, load balancer, and service mesh sidecar configuration for connection drain and endpoint removal timing.
• •The shutdown hook boundary: registered beforeExit, SIGTERM listeners, and the explicit ordering of "stop accepting" → "wait for in-flight" → "close resources" → "exit".

Diagnostic steps

1. 01Verify signal delivery: confirm SIGTERM is reaching the application process by comparing orchestration termination timestamps with the application's "received signal" log line; absence indicates an init wrapper or PID misassignment is swallowing the signal.
2. 02Verify drain invocation: confirm server.close() (or platform equivalent) is called inside the SIGTERM handler; if missing, the listener continues to accept sockets and the policy is not being enforced.
3. 03Verify drain completion condition: confirm the handler waits for the server.close() callback (or for an explicit in-flight counter to reach zero) before calling process.exit(); exiting on a timer alone violates the bounded policy.
4. 04Verify grace window sizing: compare the longest observed request duration against terminationGracePeriodSeconds; a window shorter than p99 in-flight duration will force-kill legitimate work.
5. 05Verify readiness gating: confirm the readiness probe flips to "not ready" before SIGTERM is sent (preStop hook) so the load balancer stops routing new traffic; otherwise new requests keep arriving during the drain.
6. 06Verify keep-alive drain: confirm idle keep-alive connections are closed by the server during drain; lingering idle sockets can keep the close() callback pending past the grace window.
7. 07Verify child process compliance: if workers or background queues exist, confirm they also honor the shutdown signal; uncoordinated children can hold the event loop open or write responses after the listener is closed.
8. 08Distinguish orchestrator kill from application bug: a SIGKILL after grace expiry implies the application did not exit in time, which is a policy-length or hook-ordering defect, not a signal-delivery defect.

Common mistakes

• •Treating any single clean shutdown under no load as proof of graceful shutdown; the policy must hold under in-flight requests, not only when the listener is idle.
• •Increasing terminationGracePeriodSeconds without first verifying the application actually drains, which masks the real defect behind a longer wall clock.
• •Relying on process.exit() inside the SIGTERM handler without calling server.close() first, which terminates the event loop before in-flight responses can flush.
• •Letting the readiness probe remain green until the moment the container terminates, so the load balancer keeps sending new requests into a draining instance.
• •Wrapping the application in an init process that does not forward SIGTERM, leaving the application unaware that shutdown has begun.
• •Closing the listener but not draining keep-alive sockets, so server.close() never calls back within the grace window and the orchestrator escalates to SIGKILL.

Safe fixes

• •If SIGTERM is not reaching the application: ensure the container command runs the application as PID 1 directly, or use an init wrapper that forwards SIGTERM to the child, and re-verify the "received signal" log line appears.
• •If the drain is missing: register a SIGTERM handler that calls server.close() (or equivalent), then awaits the close callback or an in-flight counter reaching zero, and only then calls process.exit(); re-verify the sequence in logs.
• •If the grace window is too short: raise terminationGracePeriodSeconds only after confirming p99 in-flight request duration plus a safety margin is below the new window; re-verify with a load-injected shutdown test.
• •If readiness is not gated: add a preStop hook that flips readiness to "not ready" and sleeps briefly before SIGTERM, so the load balancer stops routing; re-verify by observing new-connection counts drop to zero before drain completes.
• •If keep-alive sockets linger: close idle keep-alive sockets explicitly during the drain phase so server.close() can complete; re-verify the close callback fires within the budget.
• •If background workers ignore shutdown: propagate the shutdown signal to workers or queue drainers and wait for their quiescence before exiting; re-verify by checking worker shutdown logs align with the parent drain.

Prove the fix

1. 01During a forced restart with synthetic load, observe zero new 5xx responses and zero connection resets on requests that were in-flight before SIGTERM, and confirm all such requests return their normal 2xx response.
2. 02Observe the application logs in order: SIGTERM received → server.close() called → "in-flight count: N" → "in-flight count: 0" → process.exit(0), with the drain interval within the configured grace window.
3. 03Observe the readiness probe transition to "not ready" before SIGTERM and the load balancer stop sending new connections before the drain begins, verified via upstream access logs.
4. 04Repeat the verification after any change to the shutdown handler, signal wrapper, deployment manifest, or upstream drain configuration, and treat a single failed run as a regression.

Prevention and next steps

• •Treat graceful shutdown as a tested contract: include a load-injected shutdown test in CI that asserts no in-flight request is reset and the process exits within the grace window.
• •Keep the shutdown sequence explicit and ordered in code: stop accepting, await in-flight zero, close resources, exit; document each step and its corresponding log line.
• •Right-size terminationGracePeriodSeconds against measured p99 in-flight duration plus a safety margin, and revisit whenever request latency profiles change.
• •Gate readiness on drain readiness, not on process liveness, so the load balancer removes the endpoint before SIGTERM and adds it back only after a clean start.
• •Avoid init wrappers or supervisors that swallow SIGTERM, and verify signal forwarding with an explicit log entry on every restart.

Safe commands and checks

ps -o pid,ppid,stat,cmd -p <pid>  # confirm <pid> is the application PID 1 and identify any parent init wrapper that may intercept SIGTERM.
kill -TERM <pid>  # deliver SIGTERM manually to a non-production instance to observe the shutdown sequence without a deploy.
ss -tan state established '( sport = :<port> )'  # enumerate active established sockets on <port> at the moment of SIGTERM to measure in-flight load.
kill -0 <pid>  # probe liveness of <pid> during the drain window without sending a signal, to confirm the process is still draining rather than already exited.
grep -nE 'SIGTERM|shutdown|drain|close|in[-_ ]flight' <logfile>  # extract the ordered shutdown log lines from <logfile> for the most recent restart.
awk 'NR==FNR{ts[$1]=1; next} {if(ts[$1]) print}' <sigterm_timestamps> <access_log>  # correlate upstream access log entries with SIGTERM timestamps from <sigterm_timestamps> to detect resets during the drain.