Queue poison message: isolate the payload that repeatedly fails

The symptoms

• •One job identifier appears in worker error logs across many consecutive attempts, far more often than any other job, indicating retry exhaustion around a single payload rather than a class-wide bug.
• •Queue latency, completion count, or stalled-job count rises while throughput stalls because the same message is repeatedly re-delivered and consumes worker time without reaching completed or failed terminal states.
• •Retry or backoff config shows near-maximum attempt counts on a small set of jobs while the rest of the queue processes normally, signalling per-message failure rather than systemic worker failure.

Likely causes

• •Malformed or schema-invalid payload that the handler rejects deterministically on every retry, so the worker never produces a terminal state and the broker keeps re-delivering the same job.
• •Missing or unreachable downstream dependency called from the handler, producing a repeated transient error that, when retries are configured, consumes the full retry budget and stalls the worker that holds the lock.
• •Worker process crash, OOM, or unclean shutdown while the job is in-flight, leaving the queue's lock active and causing the broker to mark the job stalled and re-queue it for another worker, repeating until recovery.
• •Non-idempotent handler logic interacting with a payload that triggers a side effect the worker cannot roll back, so the failure persists across retries and blocks the queue worker pool.

First ten minutes

1. 01Confirm queue health and failed attempt counts using only the broker's read-only summary endpoint or dashboard panel, recording total attempts, max attempts, and stalled count without issuing any modify or delete call.
2. 02Identify the poison job by sorting the queue's failed-jobs or active-jobs listing by attempt count; the candidate is the job with the highest attempt counter and no completed timestamp.
3. 03Read the job's last error event and stack trace from the queue UI or a read-only inspect command, and record the exact error class, message, and the failing handler line as recorded evidence.
4. 04Capture the job payload shape (keys, nesting depth, and approximate size) from a read-only fetch, then compare against the known-good samples from neighbouring successful jobs in the same queue.

Evidence to collect

• •Job identifier (id or jobId), the queue name, and the current attempt count versus the configured max attempts, taken from the broker's read-only listing.
• •The exact error message and stack trace from the worker's last failed event for that job, including handler module path and line number if exposed by the queue framework.
• •The raw payload associated with the failing job, including any headers, correlation IDs, or metadata, captured read-only and not replayed.
• •Stalled-job events, worker lock state, and the timestamp of the last activity for that job id, used to distinguish handler errors from worker-side stalls.

Where to look

• •The queue framework's failed-jobs, stalled-jobs, and job-detail surfaces, since poison messages accumulate there before any application log.
• •The worker's structured log stream, filtered to the captured job identifier, looking for repeated handler exceptions with the same error class across attempts.
• •The application's input validation or parsing layer entry points invoked by the handler, where malformed or schema-violating payloads first surface as a deterministic error.
• •The broker's stalled-job accounting and worker lock dashboard, to confirm whether the lock holder is still alive or whether the previous worker exited uncleanly.

Diagnostic steps

1. 01Decide whether the failing job id has attempt count at or near max attempts and no completed-at timestamp; if so, classify it as the poison candidate and proceed to payload-level checks.
2. 02Compare the error class across attempts: deterministic errors that repeat verbatim point at payload or schema issues, while oscillating or time-out errors point at downstream or worker-stall causes.
3. 03Inspect the candidate's payload against the schema expected at the handler entry point, using a read-only validator (dry-run mode) and recording every validation violation as scoped evidence.
4. 04Check stalled-job records for the captured job id to rule out a worker-side hold; if stalled events coincide with attempt increments, the cause is worker health rather than payload content.
5. 05Re-run the candidate payload against a non-mutating replay tool in an isolated environment (no side effects, no external calls) and confirm whether the failure reproduces without the live broker.

Common mistakes

• •Pausing the entire queue or deleting the queue outright to resolve one bad job, which discards unrelated healthy messages and does not prove the payload was the cause.
• •Increasing the retry or backoff budget speculatively, which lengthens the time the same poison message occupies a worker without changing the underlying failure.
• •Treating repeated retries as proof of a "flaky network" without first confirming the error class, message, and payload shape are stable across attempts.
• •Replaying the suspected payload into production against the live downstream systems, which risks triggering the same side effects that caused the original failure cascade.

Safe fixes

• •Conditional on confirmed malformed payload: move the failing job id to a dedicated dead-letter or quarantine queue using the queue framework's move-job capability, rather than deleting it, so the payload is preserved for forensic review.
• •Conditional on confirmed downstream unavailability: pause only the consumer for this specific queue (not other queues) once the offending job id has been moved aside, then resume after the dependency health check returns to nominal.
• •Conditional on confirmed worker stall: address worker process health (restart the affected worker, address OOM, fix unclean shutdown) before re-queueing, and verify stalled count returns to zero for that job id.
• •After the immediate cause is addressed, add a schema validation step at the handler entry point so future payloads with the same shape are rejected before they consume retry budget.

Prove the fix

1. 01Queue stall count returns to baseline within one monitoring interval and the candidate job id is no longer present in active, waiting, or stalled surfaces after the move-to-quarantine step.
2. 02The handler's per-minute throughput returns to the pre-incident baseline for the same queue, demonstrating the poison message is no longer occupying worker capacity.
3. 03A repeated re-injection of the captured payload into the non-mutating replay tool produces the same validation error each time, proving the diagnosis and not a transient cause.
4. 04No new jobs accumulate attempts at or near the configured maximum for at least one full retry cycle after the fix, confirming the underlying cause is bounded.

Prevention and next steps

• •Validate every job payload against a schema at handler entry, and route validation failures directly to a quarantine queue rather than the retry path.
• •Cap the worst-case worker time spent on a single job id, and surface a metric for jobs that cross a high attempt threshold so poison messages are visible before retry exhaustion.
• •Keep worker health, memory, and shutdown signals observable and tied to the queue's stalled-job metric, so worker-side stalls are distinguished from payload-side failures.
• •Document the move-to-quarantine procedure and the read-only diagnostic commands so the next on-call can repeat the isolation without speculative changes.

Safe commands and checks

queue-cli failed list --queue <queue-name> --sort attempts:desc --limit 10 # read-only; list top failed jobs by attempt count
queue-cli job show --queue <queue-name> --id <job-id> # read-only; inspect job detail, attempt counter, and last error
queue-cli stalled list --queue <queue-name> # read-only; confirm stalled accounting for the candidate job id
queue-cli job move --queue <queue-name> --id <job-id> --destination <quarantine-queue-name> # scoped; moves only the named job
queue-cli worker pause --queue <queue-name> # scoped; pauses this consumer only
queue-cli consumer resume --queue <queue-name> # scoped; resumes this consumer once dependency health is nominal