Debugging Queue Consumer Failures

How this failure announces itself.

• warningFailed jobs disappear - no retry, no dead-letter entry, just gone.
• warningThe same job processes twice under load or during deploys.
• warningRetries stop after provider or SDK errors that were swallowed inside the handler.

The first ten minutes — establish facts before touching code.

• 1Pick one vanished or duplicated job and reconstruct its timeline: received, side effects, ack or nack, redeliveries.
• 2Read the handler and mark the exact line where the ack happens relative to the awaited work.
• 3Compare the job's real duration against the visibility timeout.
• 4Check what the handler does with exceptions - rethrow (nack) or swallow (silent ack)?

The code and config that usually owns this bug.

• searchAck/nack ordering relative to awaits - an ack before an awaited side effect is an ack of nothing.
• searchVisibility timeout versus real processing time - long jobs must extend visibility or they get redelivered mid-work.
• searchWorker shutdown handling - deploys that kill workers mid-job are a live test of your redelivery story.
• searchError handling inside handlers - a caught-and-logged error still looks like success to the queue.

Fix the cause, then make the regression impossible.

• buildAck only after the side effect is durably complete; on error, nack or requeue explicitly.
• buildExtend visibility (heartbeat) for long jobs before the timeout lapses.
• buildRecord idempotent job completion (a processed-ids store) so redelivery after a crash is harmless.
• buildRoute poison messages to a dead-letter queue after bounded attempts instead of retrying forever or dropping them.

A fix you can't demonstrate is a guess. Close the loop.

• verifiedThrow inside the handler in a test: the job must be redelivered or land in the DLQ, never vanish.
• verifiedKill the worker mid-job: the job re-runs and completes exactly once overall.
• verifiedProcess a job slower than the visibility timeout in a test and confirm no duplicate execution.