What this usually means
When messages are stuck unacked, the root cause is almost always that a consumer has fetched messages via basic.consume but has not sent a basic.ack or basic.nack for those delivery tags. The consumer is still connected (channel is open) but is not processing—this is a consumer-side stall. Common causes: synchronous processing that blocks on I/O (database, HTTP call) without a timeout; a consumer that crashed mid-processing without a proper shutdown hook; a prefetch count that is too high, causing a single slow message to block many subsequent ones; or a channel-level exception that the client library swallowed. RabbitMQ itself does not time out unacked messages unless consumer_timeout is set (default 30 minutes in recent versions). So the messages sit forever, consuming memory and blocking queue delivery to other consumers.
The first ten minutes — establish facts before touching code.
- 1Check RabbitMQ Management UI / Queues → select the problematic queue → look at 'Unacked' count; also note 'Consumer utilisation' (should be near 100%)
- 2Run 'rabbitmqctl list_consumers' to see all consumers on the queue, their channel, and whether they are active
- 3Check RabbitMQ logs (often /var/log/rabbitmq/rabbit@host.log) for 'missed heartbeats' or 'consumer timeout' messages; these pinpoint the stalled consumer connection
- 4Inspect the consumer application logs for any unhandled exceptions, thread dumps, or stuck worker threads; look for long-running operations without timeouts
- 5Use 'rabbitmqctl list_channels' to see the channel in question; note the 'messages_unacknowledged' field and compare to prefetch count
- 6Simulate: publish a test message with a short TTL to see if it gets consumed; if not, the consumer is definitely stuck
The specific files, logs, configs, and dashboards that usually own this bug.
- searchRabbitMQ Management UI → Queues tab → specific queue → 'Unacked' and 'Consumer utilisation'
- searchRabbitMQ logs: /var/log/rabbitmq/rabbit@hostname.log (grep for 'missed heartbeats', 'channel error', 'consumer timeout')
- searchConsumer application logs: search for 'ack', 'nack', 'reject', 'timeout', 'exception' in the consumer worker threads
- searchThread dumps of the consumer process (e.g., jstack for Java, SIGQUIT for Erlang) to see if consumer threads are blocked on I/O
- searchNetwork monitoring: check for packet loss or latency between consumer and RabbitMQ (tcpdump, netstat)
- searchApplication health endpoints: check for any backpressure indicators like database connection pool exhaustion
Practical causes, not theory. These are the things you will actually find.
- warningConsumer code blocks on synchronous I/O (DB query, HTTP call) without a timeout, causing the worker to hang indefinitely
- warningPrefetch count set too high (e.g., 1000) — one slow message blocks all prefetched messages from being acked
- warningConsumer crashes silently (unhandled exception) before ack/nack, and auto-recovery reconnects without cleaning up old delivery tags
- warningChannel-level exception (e.g., invalid routing key) that the client library logs but does not close the channel, leaving unacked messages orphaned
- warningNetwork partition between consumer and RabbitMQ — consumer thinks it's connected but heartbeats are lost; RabbitMQ marks consumer as dead only after timeout
- warningConsumer using auto-ack=false but never calling basic.ack in a finally block, so exceptions skip the ack
- warningRabbitMQ node runs out of memory (memory alarm) and stops delivering messages, but consumers still hold unacked ones
Concrete fix directions. Pick the one that matches your root cause.
- buildSet a consumer_timeout in rabbitmq.conf: 'consumer_timeout = 300000' (5 minutes) to force RabbitMQ to close channels that don't ack in time
- buildReduce prefetch count to 1 or a low number (e.g., 10) to limit the number of unacked messages per consumer; this spreads load across multiple consumers
- buildWrap the message processing in a try-finally block that always acks or nacks, even on exceptions; nack with requeue=false to avoid infinite loops
- buildImplement a processing timeout per message (e.g., using a future with timeout) so that stuck messages are nacked and redelivered
- buildUse manual acknowledgements with a separate ack goroutine/thread that batches acks, reducing chance of missed acks
- buildAdd health checks that monitor unacked count and alert when it exceeds a threshold; auto-restart consumer if stuck
- buildFor long-running tasks, use a separate queue with small prefetch and multiple consumers to avoid head-of-line blocking
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, publish a burst of messages and watch the 'Unacked' count in Management UI — it should stay near zero after an initial spike
- verifiedCheck 'Consumer utilisation' — should be 100% under sustained load
- verifiedSimulate a slow consumer by adding a sleep in processing; verify that messages are nacked after the consumer timeout and redelivered
- verifiedCheck RabbitMQ logs for absence of 'missed heartbeats' or 'consumer timeout' errors
- verifiedRun a soak test: send 10,000 messages with varying processing times; confirm no unacked pileup
- verifiedMonitor consumer application logs for consistent ack logs (e.g., 'Message acked, delivery tag X')
Things that make this bug worse or harder to find.
- warningSetting consumer_timeout too low (e.g., 10 seconds) — legitimate long-running tasks will be killed; choose based on max processing time plus buffer
- warningUsing auto-ack=true for reliability-critical flows — you lose the ability to redeliver on failure
- warningIgnoring prefetch count — default is unlimited in some clients; always set an explicit value
- warningCatching exceptions in the consumer but not calling basic.nack — the message remains unacked forever
- warningAssuming restarting the consumer clears the issue — it only masks the root cause; the unacked messages will reappear if the same code path hits again
- warningNot monitoring the unacked metric in production — it's the single best early indicator of consumer health
The Silent Consumer: 50K Unacked Messages After a Database Outage
Timeline
- 14:02Monitoring alerts: queue 'order.processing' unacked count spikes from 200 to 12,000 in 3 minutes
- 14:03Management UI shows unacked=12,345, ready=5; consumer utilisation=0% for the only consumer
- 14:04I check RabbitMQ logs; see 'missed heartbeats from consumer on channel <1.2.3.4:5672>' repeated 5 times
- 14:05Check consumer pod logs; find 'WARN - BasicReject - requeue=false' but no ack after that; also see 'DB connection timeout' errors
- 14:06Thread dump of consumer pod: 10 worker threads all blocked on 'java.net.SocketInputStream.socketRead0' (DB connection pool exhausted)
- 14:08I restart the consumer pod; unacked count starts dropping but stays high as messages are redelivered
- 14:10Identify root cause: consumer code catches DB exception, logs it, calls channel.basicReject (requeue=false) but does not return — falls through to ack later that never happens
- 14:12Deploy hotfix: wrap processing in try-finally that always acks; also reduce prefetch from 250 to 10
- 14:15Unacked count drops to near zero; consumer utilisation returns to 100%
The alert hit at 14:02 — unacked messages for the order processing queue were climbing exponentially. By the time I opened the management UI, we had over 12,000 unacked messages and a single consumer with 0% utilisation. The ready queue was nearly empty, so all messages were stuck in the consumer's buffer. My immediate thought: the consumer is alive but not acking.
I checked RabbitMQ logs first — missed heartbeats from the consumer's IP. That confirmed the consumer wasn't sending heartbeats, likely because its threads were blocked. The consumer pod logs had a warning: 'BasicReject - requeue=false' followed by silence — no ack. Under that, I saw 'DB connection timeout' from the PostgreSQL pool. That was the clue: the consumer tried to save the order, the DB was slow, the pool exhausted, and the thread threw an exception. But the exception handler called basicReject and then returned? Actually, it didn't return — it was in a catch block that logged and then fell through to a later ack that never executed because the thread never got there.
I took a thread dump: all 10 consumer threads were blocked on socketRead0 — waiting for a DB connection that would never come because the pool was exhausted. With prefetch=250, each thread had 25 unacked messages. That's 250 unacked per consumer, times 10 threads = 2,500 unacked from this pod alone. But the queue had one consumer, so all 12,000 unacked were from this pod. The fix was simple: wrap the entire processing in a try-finally that acks the message. Also reduce prefetch to 10 to limit blast radius. I deployed the hotfix, and within 3 minutes unacked dropped to zero. The lesson: always ack in finally, and monitor unacked as a leading indicator.
Root cause
Consumer code had a catch block that called basicReject but did not return, so the message was rejected but the thread continued; when the DB timeout exception occurred, the thread was stuck on socketRead0, never reaching the ack. With high prefetch (250), all threads blocked, leaving thousands of messages unacked.
The fix
Added try-finally block around message processing that always calls channel.basicAck (or basicNack with requeue=false on exception). Reduced prefetch from 250 to 10 to limit unacked per consumer. Set consumer_timeout=300000 in rabbitmq.conf to force close stuck channels.
The lesson
Never assume an exception handler will exit the processing flow. Always ack/nack in a finally block. Monitor unacked count and consumer utilisation as first-line health metrics. Keep prefetch low enough that a single slow message doesn't block a large batch.
RabbitMQ prefetch (basic.qos) controls how many messages are sent to a consumer before waiting for acks. A high prefetch (like 250 or 1000) seems efficient but becomes a liability when a single message processing hangs. The consumer has already received those messages; they are all 'in flight' (unacked). If the consumer stalls, all those messages are stuck until either the consumer recovers or the connection times out.
The math: with prefetch=250 and 10 consumer threads, each thread can have up to 25 unacked messages. If one thread blocks on a database call, it holds 25 messages hostage. The other threads might continue, but if the blocking condition is global (e.g., DB pool exhaustion), all threads block, and you have 2500 unacked messages. This is exactly what happened in the sample incident.
The fix: set prefetch to 1 for critical queues where processing time varies. This ensures that a slow message only blocks itself. For high-throughput queues, prefetch of 10-50 is safer. Monitor consumer utilisation: if it drops below 100%, you have a prefetch or stall problem.
In RabbitMQ 3.8+, the consumer_timeout configuration option (default: 30 minutes) tells the broker to close a channel if no ack/nack is received within that time for any delivery tag. This is your last line of defense. Without it, a stalled consumer can hold messages indefinitely.
Set consumer_timeout in rabbitmq.conf: 'consumer_timeout = 300000' (5 minutes). Choose a value slightly above your maximum expected processing time. If a message takes 4 minutes normally, set timeout to 5 minutes. Too low, and you'll kill legitimate long-running tasks.
When the timeout triggers, RabbitMQ logs 'consumer timeout' and closes the channel. All unacked messages on that channel are requeued (unless they were already rejected with requeue=false). This prevents permanent message loss but can cause redelivery storms if the consumer is slow. Combine with dead letter queues for messages that exceed retry counts.
A consumer that is stuck on I/O might still send heartbeats if the network stack is responsive. But if the thread is blocked, it may not process incoming heartbeats from the broker, leading to 'missed heartbeats' logs. In the sample incident, the missed heartbeats were the first clue.
Monitor the 'consumer utilisation' metric in RabbitMQ Management UI or via Prometheus (rabbitmq_queue_consumer_utilisation). A value below 100% means the consumer is not keeping up; if it drops to 0%, the consumer is likely stuck. Set alerts on this metric.
Also monitor 'messages_unacknowledged' per channel (rabbitmqctl list_channels). A sudden spike with no corresponding increase in 'ready' indicates a consumer stall. Use a tool like the RabbitMQ Prometheus exporter to track these in your monitoring system.
In any consumer that uses manual ack (autoAck=false), the ack/nack call must be in a finally block. If you put it after the try-catch, an exception that doesn't return will skip it. Even if you catch the exception and call nack, if you then fall through to the ack, you'll ack a message that you already nacked — causing a double ack (which RabbitMQ ignores, but still bad practice).
The pattern: try { process(); channel.basicAck(); } catch (Exception e) { channel.basicNack(requeue=false); } finally { // no ack here, but ensure cleanup } Actually, the cleanest is to have a single point of ack/nack in a finally block based on a success flag. For example:
boolean success = false; try { process(); success = true; } catch (Exception e) { log.error(...); } finally { if (success) channel.basicAck(); else channel.basicNack(requeue=false); }
This guarantees that every message is either acked or nacked exactly once, regardless of exceptions. Test this with a forced exception to ensure it works.
Frequently asked questions
What does 'unacked' mean in RabbitMQ?
When a consumer fetches a message via basic.consume with autoAck=false, the message is marked as 'unacknowledged' (unacked) until the consumer sends a basic.ack or basic.nack. The message is still in the queue but not available for other consumers. If the consumer never acks or nacks, the message remains unacked forever (unless consumer_timeout or connection loss occurs).
How do I view unacked messages per consumer?
Use 'rabbitmqctl list_consumers' to see each consumer's channel and queue, and 'rabbitmqctl list_channels' to see per-channel unacked count. In the Management UI, click on a queue and look at the 'Unacked' number. You can also use the Prometheus exporter: 'rabbitmq_queue_messages_unacked'.
Should I set consumer_timeout to 0 to disable it?
No. Setting consumer_timeout to 0 disables the timeout, meaning a stalled consumer will hold messages indefinitely. Always set a reasonable timeout (e.g., 5-30 minutes) based on your processing time. If you have long-running tasks, consider increasing the timeout or redesigning to use a separate queue with smaller messages.
Can a high prefetch cause messages to be stuck unacked?
Yes, indirectly. High prefetch means a consumer fetches many messages at once. If one message causes the consumer to stall (e.g., blocking I/O), all prefetched messages remain unacked until the consumer recovers or the connection times out. This amplifies the blast radius of a single slow message. Reducing prefetch limits the number of unacked messages per consumer.
What happens to unacked messages when a consumer disconnects?
When a consumer disconnects (or its channel is closed), all unacked messages on that channel are automatically requeued by RabbitMQ. They become available for redelivery to other consumers. If the messages were rejected with requeue=false, they are discarded or sent to a dead letter queue if configured.