How to verify poison messages are isolated

The symptoms

• •One specific job repeatedly transitions to stalled then re-enters the queue, while other jobs in the same queue continue to process normally.
• •Worker concurrency is saturated by a small subset of jobs that never reach the completed or failed terminal state on their own.
• •Queue depth grows despite workers being present and idle capacity being reported, because the same message id keeps being picked up.
• •Logs show repeated attempts on the same job id with no progress on its data, while sibling job ids advance to completion.

Likely causes

• •A job whose handler throws an uncaught error before any retry counter is incremented, so it loops as stalled rather than moving to a failed bucket.
• •Misconfigured lockDuration or stalledInterval such that a failing job is released back to the wait list faster than its attempts counter can advance.
• •Worker process is being killed or sandboxed between attempts without calling moveToFailed, leaving the job re-queued indefinitely.
• •Handler depends on an external dependency that is permanently unavailable for that payload, but the error is swallowed or classified as transient.

First ten minutes

1. 01Identify the offending job id by comparing stalled event counts per job in the queue's event stream; isolate the single id with the highest stalled count.
2. 02Confirm that other job ids in the same queue are progressing, to rule out a worker outage rather than a poison message.
3. 03Pull the job's data and attemptsOnFailed fields and write them down before any mutation, so you have a baseline to compare against after the fix.
4. 04Decide which isolation path applies: move-to-failed with a reason, delayed retry with backoff, or manual parking in a dead-letter set.

Evidence to collect

• •Stalled event count per job id from the BullMQ QueueEvents stream, scoped to the queue name under investigation.
• •The job's attemptsMade, attemptsOnFailed, and timestamp values at the moment of the most recent stalled transition.
• •The error or exception emitted by the worker handler for that job id, captured at the moment of failure.
• •Queue depth and completed-per-minute counters before and after the suspect job is removed, to verify throughput is restored.

Where to look

• •The BullMQ stalled jobs mechanism documented at https://docs.bullmq.io/guide/jobs/stalled, specifically the lockDuration and stalledInterval interaction.
• •The QueueEvents stalled listener for the affected queue, which emits the jobId and the previous lock holder context.
• •The job's stored record in Redis under the queue key prefix, where attemptsMade and failedReason are persisted alongside the job payload.
• •The worker's own logs for the offending job id, filtered to error and stalled lines only, to compare handler output against queue-side transitions.

Diagnostic steps

1. 01Stream QueueEvents stalled for the queue and group events by jobId; a single jobId with a count significantly above its peers is the candidate poison message.
2. 02Compare the candidate's attemptsMade against its attemptsOnFailed; if attemptsMade is not incrementing between stalled events, the handler is exiting before retry accounting runs.
3. 03Inspect the handler's error for the candidate jobId; if the error class is deterministic (parse error, schema violation, missing resource), classify it as permanent rather than transient.
4. 04Verify lockDuration is greater than the longest expected handler runtime; if lockDuration is too short, every long-running job will be flagged stalled even when it is healthy.
5. 05Check whether a separate worker is competing for the same queue; concurrent workers on the same jobId without coordination will surface as stalled churn rather than progress.

Common mistakes

• •Increasing worker count to compensate for stalls caused by a poison message, which only multiplies the redundant pick-ups of the bad job.
• •Reducing lockDuration to make stalled jobs move faster, which makes healthy long-running jobs look stalled and worsens the symptom.
• •Treating every stalled event as a retry-able failure and removing the attemptsOnFailed guardrail, which hides permanent errors as transient ones.
• •Deleting the job from the queue without recording its id and reason, leaving no audit trail for the next on-call engineer.

Safe fixes

• •If attemptsMade is incrementing but the error is deterministic, call moveToFailed on the job with the captured exception and a reason string so it is parked and stops competing for capacity.
• •If attemptsMade is not incrementing because the handler aborts early, first patch the handler to rethrow and let BullMQ's retry path count the attempt, then re-evaluate.
• •If lockDuration is shorter than observed handler runtime, raise it to a value above the p99 handler duration for that queue, then watch the stalled count for that jobId drop to zero.
• •If the payload is the root cause and cannot be fixed in code, move the job to a dedicated dead-letter set keyed by the original jobId so it is preserved for offline analysis.

Prove the fix

1. 01The offending jobId no longer appears in the QueueEvents stalled stream for the queue; stall count for that id remains at zero across one full lockDuration window.
2. 02Queue depth decreases over time and completed-per-minute returns to the pre-incident baseline, proving the worker capacity that was consumed by the poison message is released.
3. 03A regression check that re-introduces a synthetic poison payload must produce a stalled event followed by a terminal failed transition within attemptsOnFailed plus one cycle, not an unbounded loop.
4. 04Other job ids in the same queue continue to complete during the regression check, demonstrating that isolation did not block healthy traffic.

Prevention and next steps

• •Set attemptsOnFailed to a finite value per queue and alert when any jobId reaches it, so a permanent failure is escalated rather than silently retried.
• •Size lockDuration from observed handler p99 latency plus a safety margin, and review it whenever handler code paths change.
• •Wrap handlers so that any thrown error reaches BullMQ's retry accounting path, and log the error class alongside jobId for post-incident search.
• •Maintain a dead-letter destination keyed by original jobId and failed timestamp, so isolated poison messages remain inspectable rather than being silently discarded.

Safe commands and checks

node -e "const {QueueEvents}=require('bullmq');const q=new QueueEvents('<queue-name>',{connection:{host:'<redis-host>',port:<port>}});q.on('stalled',({jobId,prev})=>console.log(jobId,prev));"
node -e "const {Queue}=require('bullmq');const q=new Queue('<queue-name>',{connection:{host:'<redis-host>',port:<port>}});q.getJob('<job-id>').then(j=>console.log({attemptsMade:j&&j.attemptsMade,attemptsOnFailed:j&&j.attemptsOnFailed,failedReason:j&&j.failedReason}));"
node -e "const {Queue}=require('bullmq');const q=new Queue('<queue-name>',{connection:{host:'<redis-host>',port:<port>}});q.getJobCounts().then(c=>console.log(c));"