Queue retry checklist

The symptoms

• •Jobs repeatedly transition from active back to waiting or delayed without ever reaching completed or failed terminal state.
• •Retry counters in job metadata increment while throughput stays flat, indicating repeated re-attempts of the same payload.
• •Stalled counters increase in the BullMQ metrics stream while the worker process still appears healthy in process listings.
• •Jobs vanish from active state without a corresponding completion or failure event in the queue event log.
• •Backlog in delayed or wait queues grows monotonically with no observable drain rate during expected retry windows.

Likely causes

• •Retry policy configured with unbounded attempts or an undefined attempts field, allowing jobs to re-enter the queue indefinitely.
• •backoff strategy absent or set to immediate zero-delay, so transient failures re-enter the active state without throttling.
• •Worker lifecycle loss causing stalled jobs that BullMQ re-enqueues, masking the original error as a retry instead of a failure.
• •Locks or rate limits blocking job execution longer than the stalled interval, triggering recovery requeue that resembles retry behavior.
• •Exception swallowing inside the processor that returns success after partial work, producing retries driven by downstream consumers rather than the queue.

First ten minutes

1. 01Capture the job identifier, queue name, and current state from the queue dashboard or admin UI to anchor the investigation on a specific payload.
2. 02Read the queue events stream for the target job and note the sequence of active, waiting, delayed, and stalled transitions with their timestamps.
3. 03Inspect the job attemptsMade and attempts options fields on the job document to determine whether the configured retry ceiling has been reached.
4. 04Check worker process liveness and stalledInterval setting because BullMQ treats stalled jobs as recoverable and requeues them.
5. 05Confirm whether a dead-letter or failed queue is defined and whether terminal failures are being redirected there for inspection.

Evidence to collect

• •Job document fields: attemptsMade, attempts, backoff, delay, and stalledCount across the lifecycle of the failing job.
• •Queue event log entries showing stalled, error, failed, and completed transitions with correlation identifiers.
• •Worker configuration values for stalledInterval, lockDuration, maxStalledCount, and concurrency relative to observed load.
• •Backlog depth over time for waiting and delayed sets to distinguish bursty retries from unbounded growth.
• •Sample error stack traces from the failedEvents stream and any captured exception metadata on the job record.

Where to look

• •Queue admin UI or Redis CLI at the queue key namespace where BullMQ stores job state, stalled markers, and event entries.
• •Worker process logs at the boundary where the processor function transitions between active and completion callbacks.
• •Dead-letter or failed queue sink used to capture jobs that exhaust their retry budget.
• •Retry policy configuration files or environment variables governing attempts, backoff, and delay for the target queue.
• •Scheduler or cron entry that enqueues the affected jobs, to determine whether retry churn originates upstream.

Diagnostic steps

1. 01Compare attemptsMade against the configured attempts value; if attemptsMade equals attempts and the job remains in waiting, the retry budget is exhausted and the next state must be failed or dead-letter.
2. 02Distinguish a stalled requeue from an explicit retry by checking whether a stalled event preceded the active-to-waiting transition in the queue events stream.
3. 03Validate backoff strategy by reading the delay or backoff fields between attempts; absence of delay indicates immediate requeue which can amplify load.
4. 04Inspect lockDuration and job runtime to determine whether the lock expired before completion, which produces stalled recovery rather than a genuine retry decision.
5. 05Review processor return paths to confirm that thrown errors map to failed state and that swallowed exceptions are not returning success on partial work.
6. 06Cross-check maxStalledCount against observed stalledCount; once maxStalledCount is reached, the job should transition to failed and exit the retry loop.

Common mistakes

• •Reading repeated active transitions as retries when BullMQ stalled-job recovery is the actual cause of requeue.
• •Increasing attempts to mask unbounded retries instead of fixing the underlying error or adding backoff throttling.
• •Assuming exponential backoff is active without verifying the backoff field type and unit, since custom strategies must be registered.
• •Ignoring the failed queue and relying on waiting backlog depth, which hides terminal failures that have exited the retry loop.
• •Treating stalledCount growth as worker instability when lockDuration shorter than job runtime is the actual driver.

Safe fixes

• •If attemptsMade equals attempts with no terminal transition, wire a failed-event listener that moves exhausted jobs to a dead-letter queue for manual review.
• •If backoff is absent on transient failures, configure an explicit backoff strategy with a non-zero delay matching the failure class to throttle retry storms.
• •If stalledCount rises alongside healthy worker processes, increase lockDuration to exceed observed job runtime and reduce spurious stalled recovery.
• •If processor code returns success after partial work, ensure the processor rethrows exceptions so BullMQ records them and applies the retry policy.
• •If upstream schedulers re-enqueue the same payload, deduplicate by job identifier or add an idempotency key before permitting requeue.

Prove the fix

1. 01For a representative failing job, attemptsMade stabilizes at the configured ceiling and the job transitions to failed or to the dead-letter queue within one retry window.
2. 02Queue events stream shows a stalled transition only when a worker process is genuinely unresponsive, and active-to-waiting transitions correlate with explicit error events rather than stalled recovery.
3. 03Backlog depth for waiting and delayed sets stops growing and the drain rate matches the configured concurrency over a sustained observation window.
4. 04Backoff delay between attempts produces non-zero gaps in the active timestamps, confirming throttled retry rather than immediate requeue.
5. 05Failed queue depth remains bounded across successive enqueue cycles, indicating that terminal failures are observable and not silently re-entering the retry loop.

Prevention and next steps

• •Define attempts, backoff, and a dead-letter target as mandatory queue configuration and reject deployment of queues without these fields.
• •Alert when stalledCount per job exceeds a fraction of maxStalledCount so stalled recovery is distinguished from application-driven retries.
• •Track retry budget consumption per queue and page when attemptsMade approaches the ceiling for an unusual share of jobs.
• •Review lockDuration against the p99 job runtime whenever the processor is changed, to prevent stalled recovery masquerading as retries.
• •Periodically replay dead-letter contents to verify that terminal failures remain diagnosable rather than accumulating silently.

Safe commands and checks

redis-cli LRANGE bull:<queue>:events 0 50
redis-cli HGETALL bull:<queue>:<jobId>
redis-cli ZCARD bull:<queue>:wait
redis-cli ZCARD bull:<queue>:delayed
redis-cli ZRANGE bull:<queue>:failed 0 -1 WITHSCORES