Worker acknowledgement checklist

The symptoms

• •Application log marks a job handler as completed and returned normally, yet the same job ID re-enters the active set after worker restart or after lock TTL expiry.
• •Worker emits a stalled event for a job whose handler function finished without throwing and whose last log line was a success message.
• •Job progress events stop updating near the end of execution, but the job only transitions to completed after a noticeable delay or only after the worker is restarted.
• •Counts in the active set are non-zero while no handler is observably running, indicating work was not acknowledged at the persistence boundary before runtime completion.
• •Idempotency records show the same job processed twice across a restart window, with the second execution starting from an earlier checkpoint than the last progress event.

Likely causes

• •The worker returns from the handler before the persistence layer records completion, so a crash between return and durability commit leaves the job on the active set and eligible for stalled recovery.
• •Acknowledgement is conflated with runtime return value; the worker assumes returning from processFn is equivalent to persisting completed state, but BullMQ requires an explicit moveToCompleted or successful response handling at the queue boundary.
• •Stalled job detection and lock TTL settings cause a job to be reclaimed by another worker even though the original handler still holds the lock until process exit, conflating liveness with acknowledgement.
• •Progress events are written to application telemetry but the last moveToCompleted token was not issued, so the queue's durability record lags behind the worker's reported progress.
• •Unhandled rejection or async cleanup runs after moveToCompleted, so application logs show post-acknowledgement work that can be confused with unacknowledged work on a restart.
• •Manual retry or move operations in custom code happen at the runtime boundary rather than the queue boundary, producing double processing after a restart.

First ten minutes

1. 01Capture the job ID and queue name from the symptom; freeze any retry or scheduled restart so the active state stays observable during diagnosis.
2. 02Inspect the worker's stalled event log and queue metrics for the job ID; note the timestamp gap between last progress event and any stalled or completed transition.
3. 03Compare the worker's last emitted event for the job (progress, completed, failed) against the queue's recorded state via read-only queue introspection, not by polling the worker.
4. 04Identify the lock token or job lock associated with the job and confirm whether the worker still holds it, using only read-only queue commands rather than killing processes.
5. 05Decide the boundary in question: is the failure between handler return and persistence commit, between progress update and final ack, or between lock release and durability record?

Evidence to collect

• •Job ID, queue name, worker ID, and timestamps of last progress event, completed transition, failed transition, and stalled event if any.
• •Lock token value and lock acquisition or release timestamps visible from the queue's read-only introspection surface.
• •Sequence of BullMQ worker events (active, progress, completed, failed, stalled) for the job ID in chronological order.
• •Configuration values for lock duration, stalled interval, and max stalled count relevant to the queue under diagnosis.
• •Application-side log lines emitted inside the handler, including any explicit await job.moveToCompleted or moveToFailed calls and their result values.
• •Difference between the worker's process exit timestamp and the queue's completed transition timestamp for the same job.

Where to look

• •The boundary between the worker's process function return value and the queue persistence layer where moveToCompleted or moveToFailed is recorded.
• •The lock lifecycle boundary: lockDuration acquisition, stalled-jobs checker sweep, and lock release on completion.
• •The stalled job detection boundary in the worker's stalled interval sweep, where a still-active job can be reported stalled if its lock token is not renewed before TTL expiry.
• •The progress update boundary: each job.updateProgress call writes to the queue but does not advance the job to completed; progress events must not be mistaken for acknowledgement.
• •The retry boundary in custom retry logic, where manual requeueing at the runtime boundary rather than the queue boundary can produce duplicate processing after restart.

Diagnostic steps

1. 01Reproduce the symptom on a single named queue by enqueuing one job with a known ID and observing whether it remains in the active set after the handler returns without throwing.
2. 02Use read-only queue introspection to read the job's lock token and lock expiration time; if the worker has returned but the lock is still held, the worker has not yet crossed the durability boundary.
3. 03Compare the worker's stalled interval to the handler duration; if the handler runtime exceeds lockDuration and no progress update renews the lock, stalled detection will fire even though the handler is still running.
4. 04Inspect the handler code for the exact point where moveToCompleted is awaited versus where the process function returns; the gap between these two points is the unacknowledged durability window.
5. 05Trace whether post-return work (cleanup, metrics emission, async flushing) runs after the moveToCompleted await; if it does, a crash in that window will leave the job in completed but the application believing work is unfinished.
6. 06Verify whether the worker uses default stalled handling or a custom stalled callback; custom callbacks that requeue without checking the handler state can move jobs out of completed back into active.
7. 07Confirm whether any manual moveToCompleted or moveToFailed is performed inside the handler; duplicate calls on the same job can race with the worker's own acknowledgement path.
8. 08Decide the boundary at which acknowledgement is meant to occur; the fix depends on whether the contract is ack-on-return, ack-on-progress, or ack-on-explicit-persist.

Common mistakes

• •Treating the handler's return value or final log line as evidence of acknowledgement, when BullMQ requires an explicit completed transition at the queue boundary.
• •Conflating lock release with acknowledgement; lock release and durability commit are separate events and can be observed independently.
• •Setting lockDuration shorter than the longest expected handler runtime without increasing stalled interval or progress renewal, producing spurious stalled events on legitimate work.
• •Reading progress events as evidence that work is complete; progress only updates state and does not advance the job lifecycle.
• •Emitting application telemetry after the moveToCompleted await, so a crash in cleanup looks like unacknowledged work when in fact the queue boundary has already been crossed.
• •Re-queuing from a custom stalled callback without checking the job's last recorded state, which can move a job out of completed into active and cause double processing after restart.

Safe fixes

• •Restrict the handler so that all state changes meant to survive a restart complete before the moveToCompleted or successful response is awaited; treat the await as the durability boundary.
• •If the handler runtime can exceed lockDuration, emit job.updateProgress at a cadence shorter than lockDuration so the lock is renewed before the stalled checker sweeps.
• •Configure stalled interval and max stalled count in proportion to handler p99 duration, and document the contract so on-call engineers know when stalled events indicate a real stall.
• •Move all post-completion side effects (metrics, notifications, audit logs) inside the same critical section that ends with the moveToCompleted await, so a crash leaves a consistent state.
• •Add an explicit readiness probe inside the handler that logs the job ID and lock token immediately before the durability boundary, so the boundary is visible in telemetry.
• •Disable any custom stalled callback that performs a move operation on jobs whose last recorded state is completed, or guard it with a read of the current job state.
• •If a handler must do work after acknowledgement, wrap that work in a separate idempotent job rather than chaining it onto the awaited moveToCompleted call.
• •Treat every acknowledged job as a candidate for at-least-once processing, and make the handler idempotent against the job ID and the last persisted progress token.

Prove the fix

1. 01For a known job ID, the queue's recorded state transitions to completed within the same observation window as the handler's terminal log line, with no intervening stalled event.
2. 02After a worker restart between handler return and durability commit, the job either appears exactly once in the active set or not at all, never both, for the same job ID.
3. 03A handler whose runtime exceeds the previous lockDuration completes without producing any stalled event, and the queue's lock token is renewed at the configured cadence.
4. 04Idempotency records show at most one completed entry per job ID per worker lifetime, and the last persisted progress token equals the token emitted at the durability boundary.
5. 05Replaying the exact sequence of stalled events observed during the incident no longer reproduces the duplicate-processing pattern, and counts in the active set return to zero within the expected interval.
6. 06Any custom stalled callback is observed reading the job's current state before any move operation, and the move operation target state matches the intended lifecycle transition.

Prevention and next steps

• •Define the acknowledgement contract in code comments: what event marks the durability boundary, what events do not, and what side effects must occur before versus after that boundary.
• •Keep lockDuration, stalled interval, and progress update cadence in a single configuration object reviewed alongside handler changes.
• •Make every handler idempotent against the job ID and the last persisted progress token, so duplicate processing after restart is safe by construction.
• •Add a structured log line at the durability boundary that includes job ID, lock token, and elapsed runtime, so future on-call engineers can locate the boundary in telemetry.
• •Review any custom stalled or failed callbacks whenever the queue's stalled handling changes, to prevent re-queuing of completed jobs.

Safe commands and checks

redis-cli -h <host> -p <port> HGETALL bull:<queue-name>:<job-id>
redis-cli -h <host> -p <port> ZRANGE bull:<queue-name>:active 0 -1 WITHSCORES
redis-cli -h <host> -p <port> ZCARD bull:<queue-name>:active
redis-cli -h <host> -p <port> HGET bull:<queue-name>:<job-id> locktoken
redis-cli -h <host> -p <port> HGET bull:<queue-name>:<job-id> lockduration
redis-cli -h <host> -p <port> ZRANGE bull:<queue-name>:stalled 0 -1 WITHSCORES
redis-cli -h <host> -p <port> HGET bull:<queue-name>:<job-id> processedOn
redis-cli -h <host> -p <port> HGET bull:<queue-name>:<job-id> finishedOn