Debugging Node.js Cluster Module Worker Crashes

The `cluster` module uses `child_process.fork()` under the hood, which communicates via a Unix socket (or named pipe on Windows). The `child.send()` method serializes the message as JSON and writes it to the socket. There's a hard limit on the message size defined by `maxPayload` (default 100 MB in Node 14+). If the serialized message exceeds this, `send()` returns `false` and emits an 'error' event on the internal channel. However, the worker's event loop may not process this error if the code doesn't listen for it, leading to an unhandled exception or rejection.

In practice, many developers treat `worker.send()` as fire-and-forget. The return value is seldom checked. When the message is too large, the worker might not crash immediately — it could hang, or the message might be silently dropped, causing the master to think the worker is dead. The safest approach is to implement a size check before sending and, if the size is too large, send it in chunks or write to a shared database instead.

Before Node.js 15, unhandled promise rejections would terminate the process with a non-zero exit code. This is a common cause of worker crashes in clusters running older Node versions. The rejection might come from an async function that doesn't have a `.catch()` — for example, an Express route handler that is `async` but not wrapped in a try-catch. When the async function throws, the rejection propagates to the global scope. If no `unhandledRejection` handler is registered, the process exits.

The fix is twofold: (1) Add a global `process.on('unhandledRejection', (reason, promise) => { console.error('Unhandled Rejection at:', promise, 'reason:', reason); process.exit(1); })` — this at least logs the error before exiting. (2) Use an Express error-handling middleware or a centralized wrapper for all async routes to catch errors locally. In Node 15+, unhandled rejections are deprecated but still emit a warning; set `--unhandled-rejections=strict` to get the old behavior for debugging.

Workers that handle requests can accumulate memory over time due to closures that hold references to large objects, or to global caches that never expire. In a cluster, each worker has its own heap, so a leak in one worker won't affect others, but the worker will eventually grow until the OS kills it (OOM) or Node hits the `max-old-space-size` limit. The symptom is that the worker's RSS climbs steadily until it crashes, then the master forks a new worker, and the cycle repeats.

To diagnose, use `process.memoryUsage()` and log `heapUsed` every few seconds. If it increases linearly with request count, you have a leak. Take heap snapshots with the `heapdump` module and compare them using Chrome DevTools. Common leak sources: `req` and `res` objects kept alive by unclosed connections, event listeners that are never removed, and `setInterval` loops that allocate memory without cleanup. Set `--max-old-space-size=512` to force a crash earlier and trigger a restart, but the real fix is to identify and plug the leak.

Native addons (e.g., `bcrypt`, `sharp`, `node-canvas`) can cause segmentation faults if they access invalid memory. This typically happens after a Node.js upgrade without rebuilding the addon, or when the addon has a bug. A segfault causes the worker to exit with signal 11 (SIGSEGV), which the cluster master logs as 'worker exited with signal 11'. The process dumps core if configured.

To debug, enable core dumps and analyze with `gdb` or `llnode`. The call stack usually points to the native function. The fix is to rebuild all native modules with `npm rebuild` after every Node version upgrade. If the segfault persists, check the addon's issue tracker for known bugs and workarounds, or replace it with a pure JavaScript alternative.

The `cluster` module's IPC is event-driven and can have race conditions when workers are started or stopped rapidly. For example, if the master sends a message to a worker that is about to exit, the message might never be delivered, and the worker might not receive a 'disconnect' event properly. This can cause the master to think the worker is still alive, leading to inconsistent state.

To mitigate, always listen for 'exit' and 'disconnect' events on workers. Use `worker.isConnected()` before sending messages, but note that it can return `true` even if the worker is about to die. A robust pattern is to implement a health-check protocol: the master periodically sends a 'ping' message, and the worker must respond with 'pong' within a timeout. If no response, the master kills and restarts the worker. This covers both IPC failures and worker hangs.