Queue stall checklist

The symptoms

• •Jobs transition into the active state but never move to completed, failed, or delayed within the expected processing window.
• •The queue dashboard or metric shows a non-zero active count that stays flat across polling intervals while no new jobs are accepted.
• •Worker process appears responsive (heartbeat, metrics endpoint, or log emissions continue) but no job callbacks fire for the stuck entries.
• •Retry or delayed counters do not increment for the affected jobs, indicating the worker never invoked the failure path.
• •Removing the worker process causes the jobs to be reclaimed after the stalled-check interval rather than being released cleanly.

Likely causes

• •Worker process crashed, was OOM-killed, or lost its Redis connection without invoking the stalled-job detection mechanism described in the BullMQ stalled jobs guide.
• •Processor function is blocked on a synchronous CPU operation, an unresolved Promise, or a network call that never returns, preventing the next event loop tick from processing further.
• •Event loop is saturated by an unrelated workload, so the stalled-check timer and progress callbacks never get a chance to run.
• •Misconfigured stalledInterval or maxStalledCount values that mask or misclassify the stall, causing the worker to treat a stuck job as still healthy.
• •Redis network partition or eviction policy interfering with the lock and stalled-job bookkeeping keys referenced in the official BullMQ stalled documentation.

First ten minutes

1. 01Capture the current worker process identifier and confirm the process is still resident; a missing process indicates a hard crash rather than a soft stall and changes the diagnostic path.
2. 02Sample the Redis stalled-job bookkeeping keys to see whether the active entries carry a recent lock timestamp, which tells you if the worker is still negotiating liveness.
3. 03Inspect the worker logs for the most recent job pickup line and the timestamp of the last emitted event; a long gap confirms the stall rather than a slow job.
4. 04Confirm the queue's stalledInterval and maxStalledCount configuration against the values implied by your deployment so you can rule out a configuration mismatch early.
5. 05Record the job IDs of the currently active entries so subsequent steps can correlate Redis state, worker logs, and external signals without ambiguity.

Evidence to collect

• •Redis key state for the active job entries including the lock token and the timestamp of the last liveness signal recorded for each stalled entry.
• •Worker process status output, specifically whether the process exists, its resident memory, and whether it is consuming CPU within expected bounds.
• •Tail of the worker log filtered to the affected job IDs, with timestamps, to determine when each job last reported progress or acknowledged completion.
• •Queue configuration values for stalledInterval, maxStalledCount, lockDuration, and lockRenewTime so they can be cross-referenced with observed timing.
• •External dependency status for any downstream services the processor awaits, since unresolved awaits are a common cause of stalled jobs.

Where to look

• •The Redis instance backing the queue, focusing on the per-job lock keys and the stalled-job bookkeeping namespace described in the BullMQ stalled jobs guide.
• •The worker process boundary: stdout, stderr, structured log emitters, and the runtime event loop rather than the queue dashboard alone.
• •The processor callback boundary inside the application, where synchronous work, awaits, and external calls occur and where stalled detection first observes the absence of progress.
• •The deployment boundary: container or process supervisor records that show whether the worker was restarted, evicted, or paused around the time the stall began.
• •The configuration boundary: environment variables and configuration files that set stalledInterval, maxStalledCount, and lockDuration for the affected worker pool.

Diagnostic steps

1. 01Compare the active job timestamp in Redis against the queue's lockDuration and stalledInterval to determine whether the worker has exceeded the configured stalled window without renewal.
2. 02Differentiate a hard stall (no process, no liveness signal) from a soft stall (process present, no job progress) by checking both the worker PID and the most recent log emission for each stuck job.
3. 03If the process is missing, inspect the supervisor, orchestrator, or kernel records for crash signals and OOM events that explain the absence of a stalled-check heartbeat.
4. 04If the process is present, profile the event loop delay and identify the call frame holding the loop; unresolved awaits and CPU-bound work both prevent stalled-check timers from firing.
5. 05Verify that the stalled-job recovery path documented in the BullMQ guide actually fires by observing whether the affected jobs move from active back to waiting after the stalled window, without manual intervention.
6. 06Rule out Redis-side interference by confirming connectivity, checking recent eviction events, and validating that the stalled-job bookkeeping keys are being written within the expected cadence.

Common mistakes

• •Restarting the worker immediately without capturing the lock state and log tail, which destroys the evidence needed to distinguish a hard crash from a soft stall.
• •Increasing maxStalledCount to silence repeated stalled events rather than identifying the underlying cause of why the worker stopped acknowledging progress.
• •Assuming a high active count alone confirms a stall, when the count may reflect legitimately slow jobs whose processing time simply exceeds the stalled interval.
• •Pointing diagnostic attention only at the queue dashboard and ignoring the worker runtime, where most stalls originate from event loop saturation or unresolved awaits.
• •Conflating Redis connection loss with job stalls, even though a stalled worker that cannot reach Redis cannot emit the events that would normally mark the job as failed.

Safe fixes

• •If evidence shows the worker process is gone, address the underlying crash or OOM condition first; restarting the worker will only reproduce the stall until the root cause is resolved.
• •If the worker is present but the event loop is blocked, isolate the blocking call by reproducing the workload in a controlled environment and refactoring the processor to yield or offload the work.
• •If stalledInterval is shorter than realistic job duration, raise it to a value that comfortably exceeds the slowest expected processing time, then verify the change with a representative job run.
• •If Redis connectivity is unstable, restore the connection and verify that the stalled-job bookkeeping keys resume being updated before declaring the stall resolved.
• •If maxStalledCount is masking recurring stalls, lower it temporarily to surface the pattern, then raise it back only after the underlying cause is documented and mitigated.

Prove the fix

1. 01The active count returns to zero within the configured stalled interval once the worker resumes acknowledgement, without manual job requeueing.
2. 02A representative job processed by the same worker code path completes within the configured lockDuration, renews its lock, and moves to the completed state on first attempt.
3. 03Subsequent stalled events on the same job ID do not recur for a sustained observation window after the mitigation is applied and the worker redeployed.
4. 04The stalled-job bookkeeping keys in Redis show fresh timestamps at the expected cadence, confirming that the worker is negotiating liveness rather than appearing healthy by configuration alone.
5. 05The worker emits a progress or completion event for each processed job within the configured window, matching the documented BullMQ stalled-job recovery contract.

Prevention and next steps

• •Bound processor execution with explicit timeouts so that a runaway downstream call cannot hold a job in the active state past the stalled interval without detection.
• •Emit structured progress events from the processor at a cadence shorter than lockDuration, giving stalled detection visible signal rather than relying on absence of failure.
• •Keep stalledInterval, lockDuration, and maxStalledCount aligned with realistic job durations, and review them whenever processor logic or downstream dependencies change.
• •Monitor the active job age and the stalled-event rate as first-class signals so a stall is detected from telemetry before it is reported by downstream consumers.
• •Document and rehearse the worker recovery path so that a stall triggers a known sequence: evidence capture, root cause isolation, mitigation, and proof-of-fix verification.

Safe commands and checks

ps -p <pid> -o pid,etime,rss,cmd
redis-cli -u <redis-url> KEYS 'bull:<queue-name>:*' | head -n 50
redis-cli -u <redis-url> HGETALL bull:<queue-name>:<job-id>
redis-cli -u <redis-url> ZRANGE bull:<queue-name>:stalled 0 -1 WITHSCORES
redis-cli -u <redis-url> CLIENT LIST | grep -E 'id|addr|age'
redis-cli -u <redis-url> INFO memory | grep -E 'used_memory|maxmemory'
tail -n 200 <worker-log-path> | grep -E '<job-id>|stalled|active'
node --inspect=<port> -e "require('./worker-entrypoint')"