Background job remains running forever: find the missing completion edge

The symptoms

• •The job count in the active set stays non-zero for that job ID long after the worker started handling it, and never drops back down.
• •Queue dashboards or metrics show a steadily growing "active" or "in-flight" count while "completed" and "failed" stay flat.
• •The worker process is still running and consuming CPU or memory, but no completion log line, no success metric, and no failure log are produced for the affected job.
• •Other jobs in the same queue may process normally, isolating the issue to specific job IDs or specific code paths rather than the whole worker.
• •Because no exception was thrown, application error tracking and crash alerts never fire; only the queue or an external watchdog eventually flags the job.

Likely causes

• •The handler returns synchronously before its async work finishes, so the worker thinks the job is done while the inner promise is still pending.
• •An awaited call hangs indefinitely because the downstream resource (database connection, HTTP call, message broker subscription, child process) never resolves and has no timeout.
• •The success or completion branch is unreachable due to a logic error, while the failure branch is also never reached because errors are swallowed or logged without being propagated to the queue.
• •The worker process is alive but its event loop is blocked or starved, so it cannot send the heartbeat or completion signal that the queue expects.
• •The job stalls: the worker process dies or loses its lock, but no terminal signal is sent, leaving the job in a state described in the BullMQ stalled jobs documentation.
• •A callback-style API is being treated as if it returned a promise, so the worker exits the handler without ever invoking the done or complete callback.

First ten minutes

1. 01Identify the exact stuck job ID from the queue UI, Redis active set, or your worker logs; do not start changing code until you can name the failing job.
2. 02Confirm whether the issue affects one job, one class of jobs, or every job in the queue, because the scope usually determines whether the bug is in the handler or in the worker runtime.
3. 03Check that the worker process is still running and attached to the queue; an orphaned worker is the simplest explanation for a missing completion signal.
4. 04Grep the worker logs for the job ID and look for an explicit "completed", "failed", or "done" log line; absence of that line is the central clue.
5. 05Search for stall-related events or messages, which are the queue's own way of noticing that a worker stopped reporting back for a job.

Evidence to collect

• •The job ID, queue name, and the timestamp when the job was first seen as active.
• •Worker process PID, CPU and memory state, and the worker identifier or name registered with the queue.
• •Full handler logs scoped to the job ID, including any inner async calls, network requests, and database queries.
• •The current state of the job in the underlying store, read-only, to confirm it is still listed as active and not duplicated.
• •Any stall events or stalled-job markers emitted by the queue framework around the time the job stopped progressing.
• •Configuration values for timeouts, retries, stalled-job detection intervals, and lock durations so you can compare them with observed behavior.

Where to look

• •The worker's stdout and stderr streams, filtered to the job ID and to the time window between job start and now.
• •The underlying queue store (Redis keys for BullMQ, equivalent structures for other engines) for the active set and the stalled set, read-only.
• •The handler source file itself, especially the end of the handler function, error handling blocks, and any background promises spawned inside it.
• •Network and database logs for the time window, looking for connections that were opened but never closed by the handler.
• •The orchestrator or process manager (systemd, Docker, Kubernetes, PM2) for the worker's lifecycle events, restarts, and OOM or signal records.

Diagnostic steps

1. 01Reproduce locally with the exact same job payload and the same worker version, then trace every await and callback so you can see where control flow leaves the handler.
2. 02Add a temporary log line as the very last statement in the handler and another one inside every error branch; if the logs never fire, the handler never reaches the end.
3. 03Wrap the entire handler in a try/catch/finally and log in the finally block; a missing finally log proves the function exited through a path that does not run the completion code.
4. 04Compare the handler's async pattern (promise vs callback) against the queue client's expected interface; a mismatch is a frequent cause of a silent miss.
5. 05Inspect whether the worker still holds the lock for the job; a missing lock plus a missing completion signal points to a stalled worker as described in the BullMQ stalled jobs guide.
6. 06Check whether long-running work is exceeding the stalled-job detection interval, which can cause the queue to believe the worker has stopped progressing.
7. 07Validate that every awaited dependency has a finite timeout and that the client itself reports a clean disconnect rather than hanging on socket I/O.

Common mistakes

• •Returning a value or resolving before an inner background task finishes, assuming that returning from the handler is enough to mark the job complete.
• •Awaiting only part of the work, then falling through to the completion call while a separate fire-and-forget promise still holds the real result.
• •Logging an error inside a catch block but never rethrowing or signaling failure to the queue, so the job sits active with the worker thinking it recovered.
• •Mixing callback and promise styles, where the callback path completes the job but the promise path does not, or vice versa.
• •Treating a process-level event (SIGTERM, uncaughtException) as a normal exit, which can leave the job active because the completion signal was never sent.
• •Assuming that "the function returned" means "the job is done", when the function may have returned through a code path that does not call the queue's completion API.

Safe fixes

• •Introduce a wrapper around the handler that always calls the queue's completion API exactly once, on every exit path including thrown errors and unhandled rejections.
• •Audit every async call inside the handler and replace fire-and-forget patterns with explicit awaits so the completion signal cannot race ahead of the work.
• •Add bounded timeouts to every external dependency (HTTP, database, message broker) so a hung dependency produces a controlled failure rather than an indefinite wait.
• •Use a heartbeat or progress reporting mechanism for legitimately long jobs so the queue does not mark them as stalled and so the worker stays aware of liveness.
• •Standardize on a single async style (promise-based or callback-based) per handler, and document which completion API must be called on success and on failure.
• •Handle process signals and uncaught rejections in a single shutdown routine that drains or explicitly fails in-flight jobs before the worker exits.

Prove the fix

1. 01Re-run the same job payload and observe the job leaving the active state within a bounded time, transitioning to either completed or failed as expected.
2. 02Confirm that an explicit completion or failure log line is now emitted for that job ID, matching the handler's terminal branch.
3. 03Watch the queue metrics over the next several runs and verify that the active count returns to zero between jobs and that no stall events are emitted for the same code path.
4. 04Reproduce a forced failure (for example, by pointing the handler at an unreachable dependency) and confirm that the job still reaches a terminal failed state instead of remaining active.
5. 05Inspect the queue's own records to confirm the job is no longer present in any active or stalled set after the fix.

Prevention and next steps

• •Enforce a handler shape where the completion API is called from a single finally block, making it structurally impossible to exit without a terminal signal.
• •Add lint or static analysis rules that flag fire-and-forget async calls inside queue handlers and unhandled promise rejections.
• •Monitor active-vs-completed ratios and alert when a job stays active beyond an expected duration, so silent hangs are caught before users notice.
• •Keep queue configuration under review, especially the stalled-job detection interval and lock duration, and align them with the longest legitimate handler runtime.
• •Document a runbook for stalled jobs that points to the BullMQ stalled jobs guide and to the worker lifecycle steps where completion is expected.

Safe commands and checks

ps -o pid,etime,stat,cmd -p <pid> # inspect a suspected worker process; obtain <pid> from your process manager or `ps aux | grep worker`
grep -n "<job-id>" /var/log/worker/*.log # scope handler logs to the stuck job; replace <job-id> with the actual job identifier
redis-cli LLEN bull:<queue-name>:active # read-only check of how many jobs the BullMQ-style queue currently considers active
redis-cli ZRANGE bull:<queue-name>:stalled 0 -1 WITHSCORES # read-only inspection of stalled jobs; consult the BullMQ stalled jobs guide for the exact key naming
redis-cli HGETALL bull:<queue-name>:<job-id> # read-only view of a single job's stored fields; safe to run repeatedly while debugging
strace -p <pid> -e trace=network,write -c -t 30 # observe a worker's syscalls for a short window to see whether it is blocked on I/O; use the PID captured from the worker process inspection
node --inspect-brk=0.0.0.0:0 dist/worker.js # attach a debugger to a local worker repro to step through every exit path of the handler