Graceful-shutdown checklist

The symptoms

• •New HTTP requests are still being accepted after the process received SIGTERM, producing 200/201 responses that the orchestrator considers orphaned.
• •In-flight requests are truncated mid-response, with clients reporting connection resets or partial JSON payloads after a deploy or pod restart.
• •Background jobs, queue consumers, or cron tasks are killed mid-transaction, leaving database rows in a half-updated state or unacknowledged broker messages.
• •Database or message-broker clients log "Connection forcibly closed" or "ECONNRESET" errors against the process during the shutdown window.
• •The process exits with code 0 even though work was abandoned, or fails readiness while the load balancer keeps the endpoint in rotation.
• •Restart loops: orchestrator reports CrashLoopBackOff or repeated "exited too quickly" events because the process did not finish its drain within the terminationGracePeriodSeconds window.

Likely causes

• •No SIGTERM handler is registered; the default action terminates the event loop immediately, dropping sockets and DB pools.
• •A SIGTERM handler exists but does not call server.close(), so the listening socket keeps accepting new connections until the process exits.
• •Long-running requests, uploads, or WebSocket connections exceed the configured shutdown timeout, so the drain phase is cut short.
• •Database, cache, or queue clients are closed before in-flight work finishes, producing ECONNRESET against still-open transactions.
• •Health/readiness checks continue to return "ready" during shutdown, so the load balancer keeps routing new traffic to the draining instance.
• •Worker loops (setInterval, setImmediate, queue consumers) are not stopped, so the event loop stays active and process.exit fires despite the drain.
• •Kubernetes sends SIGKILL after terminationGracePeriodSeconds, terminating the process before the configured drain completes.

First ten minutes

1. 01Confirm the symptom window: record the timestamp of the deploy, the SIGTERM delivery time, and the process exit code from the orchestrator's lastState.terminated.exitCode field.
2. 02Read the Node.js errors reference at https://nodejs.org/api/errors.html to confirm the exact error class names (such as ERR_SERVER_NOT_RUNNING) that should appear if a shutdown handler is missing or misordered.
3. 03Search the application logs for the SIGTERM receipt line and the first "close" or "listening" event after it; the gap between them is the drain window.
4. 04Inventory the listening sockets (HTTP, HTTPS, gRPC, raw TCP) and the external clients (DB pool, Redis, RabbitMQ, Kafka) that must be closed in order; mark each as "stop-accept" or "drain-then-close".
5. 05Check the orchestrator's grace period: terminationGracePeriodSeconds in Kubernetes, StopTimeout in systemd, or the equivalent on the platform, and compare it to the longest expected request duration.
6. 06Capture a list of background workers (setInterval, queue subscribers, scheduled jobs) and identify which ones must be stopped before the event loop is allowed to exit.

Evidence to collect

• •Process exit code and termination reason from the orchestrator (lastState.terminated.reason, exitCode, finishedAt).
• •Server lifecycle log lines: "received SIGTERM", "server closing", "server closed", and the elapsed time between them.
• •Per-request log lines whose timestamps span the SIGTERM receipt, to identify truncated or abandoned requests.
• •Health-check responses recorded during the shutdown window, to confirm whether readiness flipped to "not ready" before the socket closed.
• •DB and broker client close events: pool.drain, pool.end, channel.close, consumer.cancel, with their success or error status.
• •Worker stop events: interval cleared, consumer cancelled, scheduled task flushed, with a timestamp for each.

Where to look

• •The Node.js process entry point and any bootstrap module that registers signal handlers or starts the HTTP server.
• •The shutdown module: a function wired to SIGTERM, SIGINT, message events, or process.on('beforeExit').
• •The health-check handler and the readiness/liveness probe configuration on the orchestrator.
• •The reverse proxy or load balancer routing rules, specifically the time it takes to remove a marked-not-ready instance from rotation.
• •Database, cache, and message-broker client initialisation, particularly the pool or channel objects and their close/drain methods.
• •Background worker registration sites: setInterval, setTimeout recurring tasks, queue consumers, cron schedulers, and worker_threads.

Diagnostic steps

1. 01Verify a SIGTERM handler exists: search the codebase for process.on('SIGTERM', or signal handlers on a graceful library; absence means the default abort behavior is in effect.
2. 02Inside the handler, confirm server.close() is called for every listening HTTP, HTTPS, or gRPC server; a missing call leaves the socket accepting new connections.
3. 03Confirm the handler awaits the close callback or its Promise before closing dependent clients, so in-flight requests still have a live DB pool.
4. 04Check the readiness endpoint: the SIGTERM handler should set a "shutting down" flag that the probe reads, so the load balancer stops sending new traffic immediately.
5. 05Verify the drain has a hard timeout (typically 20-30 seconds) that calls process.exit(1) if the close does not complete, to prevent SIGKILL from the orchestrator.
6. 06Confirm workers and interval timers are stopped (clearInterval, consumer.cancel, scheduler.stop) before the exit, so the event loop can drain naturally.
7. 07Read the Node.js errors reference to map any observed error class to a remediation: ErrnoException codes such as ECONNRESET, and SystemError markers indicate premature socket closure.

Common mistakes

• •Calling process.exit(0) immediately after registering the SIGTERM handler, which abandons every in-flight request.
• •Closing the DB pool before the HTTP server has finished serving the last request, producing ECONNRESET on the response.
• •Returning success from the readiness probe until the process actually exits, so the load balancer continues to send new traffic.
• •Setting terminationGracePeriodSeconds too low for the slowest legitimate request, so the orchestrator SIGKILLs before the drain completes.
• •Forgetting that setInterval keeps the event loop alive; the process will not exit naturally even after server.close() returns.
• •Listening only for SIGTERM and missing SIGINT, SIGHUP, or container-specific signals, so a different signal bypasses the drain entirely.
• •Closing the server socket without draining WebSocket connections, which Node.js documentation flags as a separate path from HTTP requests.

Safe fixes

• •Register a single SIGTERM handler that flips a "shutting down" flag, fails readiness immediately, and then awaits server.close() with a hard timeout before process.exit(1).
• •Wrap the shutdown in a Promise.race between the server close callback and a setTimeout(forceExitMs); set forceExitMs to terminationGracePeriodSeconds minus a safety margin such as 5 seconds.
• •Close external clients (DB pool, Redis, broker channel) only after the HTTP server has emitted 'close', so in-flight queries still have a live connection.
• •Clear every recurring timer and cancel every queue consumer in a tracked list before the final exit, so the event loop can become idle.
• •Make the readiness probe read the "shutting down" flag and return a non-2xx status with a small Retry-After header, so upstream stops sending new traffic.
• •Adjust terminationGracePeriodSeconds in the pod spec to match the longest expected request plus the client close duration, verified by the drain time observed in logs.

Prove the fix

1. 01Trigger a rolling restart and confirm, from the orchestrator events, that the process exited with code 0 after the close event, not via SIGKILL.
2. 02In the application logs, observe the sequence: "received SIGTERM", "readiness=not-ready", "server closing", "server closed", "pool drained", "exited"; the gap between the first and last line is the drain duration.
3. 03Compare the number of 5xx responses during the rolling restart window to the baseline; the fix is validated only if the count does not increase.
4. 04Inspect the database slow query or broker unacknowledged logs during the restart window; no transaction should be left open or message redelivered.
5. 05Verify readiness flipped to not-ready before the first new request was rejected, by checking that the load balancer recorded zero new requests to the draining instance after the flip.
6. 06Run a load test that issues a long request (set to half the grace period) and a concurrent deploy, then confirm the long request completes with a 2xx response and the process exits cleanly.

Prevention and next steps

• •Define a single shutdown() function that closes resources in a known order: readiness=not-ready, stop accepting, drain, close clients, exit; reuse it for SIGTERM, SIGINT, and health-check failures.
• •Keep the drain timeout smaller than the orchestrator's terminationGracePeriodSeconds, and surface the remaining time as a metric so drains approaching the limit are visible.
• •Add a regression test that starts the server, sends SIGTERM, and asserts the server stops accepting new connections within a fixed budget while the existing request completes.
• •Track active timers, intervals, and queue consumers in a registry that the shutdown function iterates, so new background work is automatically cleaned up.
• •Review the Node.js errors reference whenever a new error class appears during shutdown, to confirm it is expected (for example, ECONNRESET from a client that disconnected during drain) and not a leak.

Safe commands and checks

ps -o pid,etime,cmd -p <pid>
grep -nE 'SIGTERM|SIGINT|server.close|server closed|pool drained|shutting down' /var/log/<service>.log
grep -nE 'ECONNRESET|ECONNABORTED|ERR_SERVER_NOT_RUNNING' /var/log/<service>.log
node -e "console.log(process.eventNames().filter(n => n.startsWith('SIG')))"
kill -TERM <pid> && sleep 5 && ps -o pid,stat,etime,cmd -p <pid>