How to validate graceful shutdown under load

The symptoms

• •New client connections are accepted after a SIGTERM has been sent and the process has not exited.
• •Load balancer health checks still report healthy while in-flight requests are aborted or reset mid-response.
• •Process exit time after SIGTERM exceeds the configured drain timeout with active sockets still open.
• •Restart loops during deploys because the new instance is removed before readiness probe has flipped to ready.
• •In-flight requests return 5xx or connection resets after shutdown has been initiated instead of completing within the drain window.

Likely causes

• •SIGTERM handler is missing or only registered after server.listen resolves, so early requests during boot miss the signal listener.
• •Health/readiness probe is bound to liveness only, so the load balancer keeps routing traffic during shutdown because readiness never flips to not-ready.
• •Keep-alive sockets are not closed, so the drain timer waits on idle pooled connections that the client never closes.
• •Drain timeout is shorter than the slowest legitimate request, so compliant work is killed mid-flight.
• •Child workers, timers, or background queues are not stopped, so the event loop stays ref'd and process.exit never fires within the bounded window.

First ten minutes

1. 01Confirm the signal you intend to test is the one the process treats as the stop signal; Node treats SIGINT and SIGHUP differently from SIGTERM on POSIX per the process signal-events documentation.
2. 02Reproduce under load: drive synthetic traffic at a known rate above baseline and record baseline p99 latency and active-connection count before signaling.
3. 03Send one SIGTERM to the target process and timestamp it; begin observing accepted-connection counter, in-flight request counter, and readiness state.
4. 04Within the drain window, verify no new connections are accepted and no in-flight requests are aborted; if either is violated, halt and capture evidence before changing code.
5. 05If the process does not exit, list open sockets by state (ESTABLISHED, CLOSE_WAIT, IDLE) and identify which subsystem is keeping the loop alive.
6. 06Capture a structured shutdown log line that records signal received, drain started, readiness flipped, in-flight count reached zero, and process exit.

Evidence to collect

• •Process signal receipt log entry with timestamp and signal name; correlate to PID and parent supervisor.
• •Readiness probe response over time: a single transition from ready to not-ready inside the drain window, not after exit.
• •Counter for connections accepted after signal timestamp; expected value is zero.
• •Counter for in-flight requests at signal time and at process exit; expected delta is zero within the drain window.
• •Histogram of shutdown latency from SIGTERM to process exit across N runs; expected distribution is tight and below the configured upper bound.
• •Open socket snapshot by TCP state at exit; expected states are CLOSED or TIME_WAIT, not ESTABLISHED held by the server.

Where to look

• •Process supervisor boundary: how the init system (systemd, container runtime, orchestrator) delivers SIGTERM and what StopSignal and TimeoutStopSec are set to.
• •HTTP server boundary: the listen socket, the keep-alive agent, and the request handler entry and exit points; readiness vs liveness endpoint separation.
• •Event loop boundary: outstanding timers, interval handles, open child processes, unfinished database connections, and any work queue consumers that ref the loop.
• •Client boundary: load balancer drain configuration, connection idle timeout, and the keep-alive timeout on the upstream side.
• •Logging boundary: structured shutdown sequence fields including signal, readiness transition, in-flight drain count, and exit code.

Diagnostic steps

1. 01Verify the registered signal handler matches the StopSignal declared by the supervisor; mismatches are a common cause of ignored SIGTERM.
2. 02Measure the time from SIGTERM to readiness flipping not-ready; if it exceeds one health-check interval, the load balancer will keep routing new requests into the draining instance.
3. 03Compare drain timeout against observed p99 of the slowest endpoint; if drain is shorter, in-flight requests will be killed and the bug looks like a crash rather than a shutdown defect.
4. 04Inspect sockets held in ESTABLISHED at exit to distinguish keep-alive idle connections from active in-flight requests; the fix differs.
5. 05Check whether background workers or queue consumers register their own signal handlers or only the main process does; a missed handler in a worker can keep the loop alive.
6. 06Reproduce with the readiness endpoint exercised by the same load balancer that fronts production; a green local readiness response is not evidence under the real probe.
7. 07Run the test at a load level that exceeds steady-state to confirm the drain window holds under stress, not only at idle.

Common mistakes

• •Treating "process exited with code 0" as proof of graceful shutdown, when in-flight requests were actually aborted and the exit only happened after the supervisor SIGKILL.
• •Registering the SIGTERM listener after server.listen resolves, so the first deploy on a slow-booting host loses the signal.
• •Flipping readiness to not-ready only after the drain completes, which inverts the order and routes new traffic into a draining instance.
• •Sharing a single readiness endpoint between liveness and load-balancer drain, so Kubernetes or the LB cannot distinguish crashed from draining.
• •Using process.exit immediately on signal without awaiting in-flight work, which produces a fast but destructive shutdown that masquerades as graceful.

Safe fixes

• •If new connections are still accepted after SIGTERM, register the signal handler before server.listen and have it close the listening socket first, then flip readiness, then drain.
• •If readiness never flips, separate the readiness endpoint from liveness and return not-ready as soon as SIGTERM is received; keep liveness green until the process actually becomes unhealthy.
• •If keep-alive sockets stall the drain, configure the server to disable keep-alive during shutdown or close idle sockets after a short idle threshold inside the drain window.
• •If drain timeout is too short, set it to at least the p99 of the slowest endpoint observed in the last deploy window plus a safety margin, and codify that bound in the verification test.
• •If the loop is kept alive by timers or workers, stop accepting new work, cancel intervals, close DB pools, and await worker quiescence before allowing the process to exit.

Prove the fix

1. 01Under sustained load, send SIGTERM and observe zero new connections accepted from the signal timestamp onward, confirmed by a monotonically non-increasing accepted-connection counter.
2. 02Readiness probe transitions from ready to not-ready within one health-check interval of SIGTERM, and remains not-ready until process exit, confirmed by probe response logs.
3. 03In-flight request count returns to zero before process exit without any 5xx or connection reset on those requests, confirmed by request-level logs keyed by request id.
4. 04Shutdown latency from SIGTERM to exit stays below the configured bound across at least ten consecutive runs, with a tight distribution and no supervisor SIGKILL events in the supervisor journal.
5. 05A canary deploy in the target environment succeeds with zero failed requests attributable to the rolling instance during the drain window, confirmed by the load balancer access log.

Prevention and next steps

• •Treat readiness and liveness as separate endpoints and version the drain contract alongside the service so load balancer configuration can be validated.
• •Add a synthetic graceful-shutdown test to CI that signals a long-running instance under load and asserts the four observable invariants above.
• •Bound the shutdown latency in a configuration value and alert when a real shutdown exceeds it, so silent regressions in dependency close times are visible.
• •Document the signal-to-stop mapping between the supervisor and the process so deploy and runtime changes cannot drift the contract silently.

Safe commands and checks

node -e "console.log(process.platform, process.versions.node)"
kill -TERM <pid>
ps -o pid,stat,etime,comm -p <pid>
ss -tan state established '( sport = :<port> )'
ss -tan state time-wait '( sport = :<port> )' | wc -l
grep -E 'SIGTERM|drain|readiness|exit' <service-shutdown-log-path>
node -e "process.on('SIGTERM',()=>{console.log('term',Date.now())});setInterval(()=>{},1<<30)"