What this usually means
The root cause is almost always that the consumer failed to delete the message after processing, causing it to reappear (visibility timeout expired) and be retried until maxReceiveCount is hit. This can happen due to unhandled exceptions, timeouts longer than visibility timeout, or the consumer crashing mid-processing. Other causes include message size exceeding the SQS limit (256KB), invalid message attributes, or IAM policies that prevent the consumer from calling DeleteMessage. The redrive policy itself may be misconfigured, e.g., maxReceiveCount set too low (default 3) for a spikey workload.
The first ten minutes — establish facts before touching code.
- 1Check CloudWatch metric ApproximateNumberOfMessagesNotVisible for the main queue — if it's elevated, messages are stuck in-flight.
- 2Inspect the DLQ messages' attributes: run `aws sqs receive-message --queue-url <DLQ_URL> --attribute-names All` and look for ApproximateReceiveCount — >3 indicates retries exhausted.
- 3Examine consumer application logs for any `DeleteMessage` failures or uncaught exceptions around the time of DLQ growth.
- 4Verify the redrive policy: `aws sqs get-queue-attributes --queue-url <MAIN_QUEUE_URL> --attribute-names RedrivePolicy` — confirm maxReceiveCount and deadLetterTargetArn.
- 5Check if message size exceeds 256KB: use `aws sqs get-queue-attributes --queue-url <MAIN_QUEUE_URL> --attribute-names All` and compare with message size from DLQ message body.
- 6Set up a CloudWatch alarm on ApproximateNumberOfMessagesDelayed if using delay queues — messages may be expiring before processing.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchCloudWatch → SQS metrics: ApproximateNumberOfMessagesNotVisible, ApproximateAgeOfOldestMessage, NumberOfMessagesReceived vs Deleted.
- searchSQS queue attributes: RedrivePolicy, VisibilityTimeout, ReceiveMessageWaitTimeSeconds, MessageRetentionPeriod.
- searchConsumer application logs: search for 'DeleteMessage', 'unhandled exception', 'timeout', 'visibility timeout'.
- searchIAM roles/policies attached to consumer: ensure `sqs:DeleteMessage`, `sqs:ReceiveMessage`, `sqs:ChangeMessageVisibility` are allowed.
- searchAWS X-Ray traces or CloudWatch Logs Insights for consumer processing duration (compare to visibility timeout).
- searchSQS DLQ messages themselves: inspect MessageAttributes, Body (especially if it's a poison pill causing deserialization failure).
Practical causes, not theory. These are the things you will actually find.
- warningConsumer fails to call DeleteMessage after processing (uncaught exception, missing finally block).
- warningConsumer processing time exceeds the queue's VisibilityTimeout, making message reappear and be redelivered.
- warningmaxReceiveCount in RedrivePolicy is set too low (e.g., 1 or 2) for normal error handling.
- warningMessage size exceeds SQS limit (256KB) — consumer SDK throws exception before processing.
- warningMessage body or attributes are malformed (e.g., invalid JSON) causing consumer to fail immediately on every attempt.
- warningIAM role for consumer lacks permissions for `sqs:DeleteMessage` or `sqs:ReceiveMessage`.
- warningConsumer instance is overwhelmed (CPU/memory) causing processing delays and visibility timeouts.
Concrete fix directions. Pick the one that matches your root cause.
- buildAdd try-finally blocks in consumer to ensure DeleteMessage is called even on exceptions.
- buildIncrease VisibilityTimeout to accommodate the p95 processing latency (e.g., from 30s to 120s).
- buildImplement exponential backoff with ChangeMessageVisibility to extend timeout for long-running tasks.
- buildValidate message size before sending and reject oversized messages at producer side.
- buildSet maxReceiveCount appropriately (e.g., 5 for normal retries, 10 for flaky dependencies).
- buildFix IAM policy: add `sqs:DeleteMessage`, `sqs:ReceiveMessage`, `sqs:ChangeMessageVisibility` to the consumer role.
- buildCircuit-breaker or poison-pill pattern: move unprocessable messages to a separate manual-inspection queue after n failures.
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, send test messages and confirm ApproximateNumberOfMessagesNotVisible stays near zero after processing.
- verifiedCheck DLQ remains empty for several hours under normal traffic (monitor CloudWatch metric).
- verifiedSimulate a failing message (e.g., malformed JSON) and verify it is deleted after maxReceiveCount or moved to DLQ correctly.
- verifiedUse `aws sqs purge-queue --queue-url <DLQ_URL>` (careful) to reset, then observe no new DLQ entries.
- verifiedLoad test consumer with peak traffic and verify no messages go to DLQ due to timeouts.
- verifiedReview consumer logs for successful DeleteMessage calls for each received message.
Things that make this bug worse or harder to find.
- warningIncreasing maxReceiveCount without fixing the underlying consumer bug — just delays DLQ growth.
- warningSetting VisibilityTimeout too high (e.g., hours) causing slow processing and stale messages.
- warningForgetting to purge the DLQ after fixing — old messages remain, causing false alarms.
- warningIgnoring message attributes that might cause consumer failure (e.g., missing required fields).
- warningUsing a single DLQ for multiple queues without proper message routing (mixing causes confusion).
- warningNot monitoring the DLQ size — set CloudWatch alarm on ApproximateNumberOfMessagesVisible > 0.
- warningAssuming all DLQ messages are poison pills — some may be legitimate retries exhausted due to transient failures.
Steady DLQ Growth After Deploy of Updated Consumer
Timeline
- 09:15Deploy new Lambda consumer v2.3 with updated message processing logic.
- 09:30CloudWatch alarm triggers: DLQ ApproximateNumberOfMessagesVisible > 50.
- 09:35Check main queue metrics: ApproximateNumberOfMessagesNotVisible high (hundreds).
- 09:40Inspect DLQ message: ApproximateReceiveCount = 3, body is valid JSON.
- 09:45Examine Lambda logs: no DeleteMessage calls, but no error stack traces either.
- 09:50Check Lambda function timeout setting: 30 seconds, but DynamoDB write takes ~5s on average.
- 09:55Realize that the new code introduced a synchronous HTTP call to a slow endpoint (10s+), combined with DB write > 30s total, exceeding SQS visibility timeout (30s).
- 10:00Increase Lambda timeout to 60s, SQS visibility timeout to 90s, and add async HTTP call. Redeploy.
- 10:15Monitor: ApproximateNumberOfMessagesNotVisible drops to near zero, DLQ stops growing. Purge existing DLQ messages.
I was on-call when a CloudWatch alarm screamed that our DLQ had over 50 messages. The main queue had been processing fine, but the DLQ grew steadily. I checked the consumer—a Lambda function—and saw no errors in the logs. The messages looked perfectly valid. This was odd: no exceptions, yet messages ended up in the DLQ.
I looked at the DLQ message attributes and saw ApproximateReceiveCount of 3, meaning the consumer had received each message three times. The queue's maxReceiveCount was 3, so after three failures, the message went to DLQ. But why did it fail? I checked the Lambda's CloudWatch logs and saw the handler executed, but there was no call to `deleteMessage`. The function returned successfully, but the message was never deleted.
I traced the code: the new version added a synchronous HTTP call to an external API that timed out after 10 seconds. Combined with the DynamoDB write (5 seconds), the total processing time exceeded the Lambda timeout (30s) and the SQS visibility timeout (30s). The Lambda was killed, but the message became visible again, got redelivered, and the cycle repeated until maxReceiveCount. I increased both timeouts and made the HTTP call asynchronous. The DLQ stopped growing immediately. Lesson: always monitor processing time vs. visibility timeout, and ensure DeleteMessage is called even on failure.
Root cause
Processing time exceeded SQS visibility timeout due to a new synchronous HTTP call, causing message reappearance and retries until maxReceiveCount was exhausted, with no DeleteMessage call because the Lambda timed out.
The fix
Increased Lambda timeout from 30s to 60s, SQS visibility timeout from 30s to 90s, and refactored HTTP call to async. Also added a catch block to ensure DeleteMessage is called on non-timeout exceptions.
The lesson
Always ensure consumer processing latency is well below the visibility timeout. Monitor ApproximateNumberOfMessagesNotVisible as an early indicator. Add explicit DeleteMessage in a finally block.
The redrive policy is a JSON object attached to the source queue: `{"deadLetterTargetArn": "arn:aws:sqs:...", "maxReceiveCount": 5}`. The maxReceiveCount determines how many times a message can be received before being moved. The counter increments per ReceiveMessage call, not per processing attempt — if the consumer calls ReceiveMessage but never processes, it still counts.
Key nuance: If the consumer calls ChangeMessageVisibility to extend the timeout, the receive count does NOT reset. The only way to reset is to delete the message. Some developers mistakenly think extending visibility resets the retry counter — it does not. Also, when a message is moved to DLQ, the original message is deleted from the source queue, so you won't see it in both places.
The most common cause for DLQ routing is the consumer taking longer than the visibility timeout. Use CloudWatch metric `ApproximateNumberOfMessagesNotVisible` — if this is high, messages are stuck in-flight. Then check `ApproximateAgeOfOldestMessage` — if it's high, those messages are old and likely timing out repeatedly.
To pinpoint, set a CloudWatch Logs Insights query on consumer logs: `filter @message like /processing time/ | stats avg(@duration) by bin(5m)`. Compare p95 duration to the VisibilityTimeout value. Also check Lambda duration in CloudWatch (max, p99). If the consumer runs on EC2, use `htop` and monitor request duration metrics in app server logs.
A poison pill is a message that causes the consumer to fail every time it's processed, often due to malformed content (e.g., invalid JSON, missing required fields, or data that triggers a bug). The consumer crashes or throws an exception, never calling DeleteMessage. After maxReceiveCount, it goes to DLQ.
To diagnose, inspect the DLQ message body with `aws sqs receive-message --queue-url <DLQ_URL> --max-number-of-messages 10`. Use `jq .Body` to see the payload. If the body is valid but causes a specific error, reproduce locally. A pattern I use: forward a sample DLQ message to a test queue, process it with verbose logging, and catch the exact exception. Then fix the consumer to handle that edge case (e.g., validation before processing) and delete the message properly.
If the consumer lacks `sqs:DeleteMessage` permission, the SDK call fails silently (depending on SDK version). I've seen cases where the consumer processed the message successfully but the delete call returned an access denied error that was swallowed by a generic catch block. The message becomes visible again after visibility timeout.
Check the consumer's IAM role policy. The minimum needed: `sqs:ReceiveMessage`, `sqs:DeleteMessage`, `sqs:ChangeMessageVisibility` (if you extend timeouts). Use IAM Access Analyzer to validate. Also check if the policy uses a resource ARN that matches the queue. A common mistake: using `*` resource but queue ARN has different account or region.
Never set maxReceiveCount to 1 unless you want every failed message to go to DLQ immediately. For most systems, 3–5 is reasonable. Also ensure the DLQ has sufficient retention period (default 4 days, but you might want 14 to have time to investigate).
Avoid using the same DLQ for multiple source queues unless you add a message attribute to identify origin. Otherwise, you can't tell which queue the message came from. Also, if the DLQ itself has a redrive policy, messages can loop indefinitely — do not set a redrive policy on a DLQ. Finally, remember that DLQ messages are billed separately; monitor costs if DLQ grows large.
Frequently asked questions
Why are messages going to DLQ when my consumer logs show successful processing?
This often happens when the consumer processes the message but fails to call DeleteMessage (e.g., missing finally block, or exception after processing but before delete). The message reappears after visibility timeout, gets redelivered, and eventually maxReceiveCount is exceeded. Check your consumer code for proper deletion and ensure no exceptions are swallowed after processing.
Can a message go to DLQ without being received by the consumer?
No — a message is only moved to DLQ after it has been received (ReceiveMessage called) at least maxReceiveCount times. If you see DLQ growth but no consumer activity, check if there are multiple consumers (e.g., Lambda with reserved concurrency) or if the visibility timeout is very short, causing rapid redelivery. Also verify that the consumer is indeed calling ReceiveMessage.
How does message size affect DLQ routing?
SQS maximum message size is 256KB (including body and attributes). If a message exceeds this, the SendMessage call fails at the producer side, so it never reaches the queue. However, if the consumer SDK fails to process a message due to size (e.g., deserialization error), the message will be retried and eventually go to DLQ. Check the message body for oversized payloads by inspecting the DLQ message.
What is the difference between ApproximateNumberOfMessagesNotVisible and ApproximateNumberOfMessagesDelayed?
ApproximateNumberOfMessagesNotVisible counts messages that are currently being processed (received but not yet deleted or timed out). ApproximateNumberOfMessagesDelayed counts messages that are in a delay queue (not yet available for processing). If you see high NotVisible, messages are stuck in-flight; if high Delayed, messages are intentionally delayed. Both can contribute to DLQ indirectly.
Should I set maxReceiveCount to a very high number to avoid DLQ?
No — that defeats the purpose of a DLQ. A high maxReceiveCount (e.g., 100) means a poison pill will be retried many times, wasting resources and delaying processing of good messages. Set it high enough for transient failures (5–10) but low enough to move problematic messages quickly. Also implement exponential backoff on visibility timeout to avoid flooding the consumer.