Queue latency rises while depth stays flat: find slow consumers or hidden partitions

The symptoms

• •Age-of-oldest-message metric climbs steadily even though queue depth and waiting count are roughly constant across dashboards
• •Workers show no errors and no crash, yet completed-per-second falls below publish-per-second during the same window
• •Stalled-job counters or lock-duration warnings appear intermittently even though jobs eventually finish
• •A subset of jobs repeatedly takes much longer than peers of identical shape, suggesting a partition or shard hot-spot rather than uniform slowness
• •Latency spikes correlate with batch size, payload size, or a specific key/routing value rather than wall-clock time

Likely causes

• •Per-job processing time has grown because of a downstream dependency slowdown, so each message sits longer in the active/visible state even at steady depth
• •Workers are losing their lock on long-running jobs and the jobs are being moved back to the wait/ready set, which BullMQ flags as stalled and re-queues for retry
• •A hidden partition or shard exists where one routing key, partition, or worker subset absorbs a disproportionate share of the load while totals look balanced
• •Concurrency is too low for current per-job duration, so the queue acts like a single-channel pipe regardless of message count
• •Connection pool, event-loop, or thread saturation is throttling consumption so messages wait in the broker rather than the in-memory list

First ten minutes

1. 01Capture the current values for age-of-oldest-message, ready count, active count, delayed count, and completed-per-second from your queue dashboard; record the window so the next reading is comparable
2. 02Confirm that depth is genuinely flat and not just slowly climbing by computing the delta over a 5-minute window and comparing it to the age-of-oldest slope
3. 03Check the stalled-job metric and any stalled-events counter in BullMQ's QueueEvents stream over the same window; non-zero values point to lock-loss rather than uniform slowness
4. 04Compare publish-per-second versus completed-per-second; if completed lags publish, the system is losing throughput even at steady depth, which is the signature of the failure mode described
5. 05Group recent jobs by a likely partition key (user id, tenant id, routing key, partition id) and look for one group with disproportionate average duration or stalled count

Evidence to collect

• •Time series of age-of-oldest-message, ready, active, delayed, and completed counters over the same window
• •Stalled-events log entries and lock-duration fields for the affected worker pool during the spike window
• •Per-key duration distribution computed from job timestamps or completion events, sorted by mean and p95
• •Worker concurrency settings versus observed active job count at the same timestamp
• •Any recent change to consumer code, downstream dependency, batch size, or lock timeout that overlaps the onset of the latency rise

Where to look

• •BullMQ QueueEvents stream for stalled and failed events tied to job ids in the affected window
• •BullMQ Queue metrics endpoint or dashboard for active, waiting, completed, failed, and stalled counters
• •BullMQ stalled-jobs documentation page for the official definition of stalled, the lockDuration setting, and the stalledInterval scan behaviour
• •The boundary between the queue broker and the worker process, where lock renewal and heartbeat timestamps are recorded
• •The boundary between the worker and any downstream service it calls, where per-call latency must be separated from queue-side latency

Diagnostic steps

1. 01Plot age-of-oldest-message against ready count over the same window; a rising age with flat ready depth is the canonical signature of slow processing, not backpressure from new arrivals
2. 02Read the stalled-jobs BullMQ documentation to confirm the definition and default scan interval, then check your stalledEvents counter and lockDuration setting against that contract
3. 03If stalled count is non-zero, treat the cause as lock loss: identify the worker ids, their concurrency, and their host-level CPU/event-loop saturation during the spike
4. 04If stalled count is zero, compute per-key mean and p95 processing time and look for one partition key that dominates the slow tail; this points to a hidden partition rather than uniform slowdown
5. 05Compare observed active job count against the configured worker concurrency; if active is persistently at the concurrency ceiling while depth is flat, raise concurrency as a controlled experiment and re-measure
6. 06Separate downstream latency from queue-side latency by adding a timestamp at job pickup and another at handler return; a queue-side stall points to stalled-jobs mechanics, a downstream stall points to dependency slowness

Common mistakes

• •Concluding that latency will resolve once depth catches up, when in fact steady depth with rising age means throughput is below arrival rate right now, not just historically
• •Increasing concurrency without first distinguishing stalled/lock-loss from per-job slowness, which can amplify event-loop pressure and worsen the stall counter
• •Reading average processing time across all jobs and missing a single hot partition that is dragging the age-of-oldest metric while averages look healthy
• •Ignoring the lockDuration/stalledInterval contract and assuming a long job is simply slow, when BullMQ has already marked it stalled and re-queued it for retry
• •Trusting queue depth as the only health signal and missing the divergence between publish-per-second and completed-per-second during the same window

Safe fixes

• •If stalled-events are non-zero and the worker is event-loop or thread bound, shorten per-job work into smaller units or move heavy work off the main loop so lock renewal stays inside lockDuration, as described in the BullMQ stalled guide
• •If one partition or routing key dominates the slow tail, add a sharding key, rebalance, or isolate the hot key into its own worker pool so the hidden partition stops starving the rest
• •If active jobs are pinned at the concurrency ceiling while age keeps rising, raise concurrency by one increment and re-measure age-of-oldest over the next 5-minute window before raising again
• •If downstream dependency is the dominant cost, add a timeout, circuit breaker, and per-call latency log so the queue-side age-of-oldest metric can be compared against dependency latency directly
• •If none of the above is supported by the evidence, keep configuration unchanged and continue collecting per-key duration and stalled counters rather than guessing

Prove the fix

1. 01Age-of-oldest-message returns to a stable band over a 30-minute window while ready count remains in its normal range, proving the slowdown has resolved rather than been masked by a depth change
2. 02Completed-per-second equals or exceeds publish-per-second across the same window, so the queue is no longer losing throughput at steady depth
3. 03Stalled-events counter drops to zero (or back to its pre-incident baseline) for the same window, confirming that lock-loss was the proximate cause when that branch was chosen
4. 04If the chosen branch was partition rebalancing, per-key p95 processing time converges across the previously hot key and a sampled normal key, showing the hidden partition has been flattened
5. 05A short rollback note is recorded describing which evidence triggered the change, so the fix can be reverted if a regression returns the same signature within the next monitoring cycle

Prevention and next steps

• •Track age-of-oldest-message alongside ready and active counts on the same dashboard so the "rising age, flat depth" signature is visible before it becomes an incident
• •Alert on stalled-events rate and on the divergence between publish-per-second and completed-per-second, not on raw queue depth alone
• •Set lockDuration and stalledInterval against measured p99 handler duration so long but legitimate work does not get misclassified as stalled, per the BullMQ stalled guide
• •Periodically audit partition or routing-key distribution to detect hot keys early, and review worker concurrency against active-job ceiling during load tests

Safe commands and checks

echo "Observe only: read stalled-events from your BullMQ QueueEvents stream (no live URL exposed here)"
echo "Compute per-key duration: aggregate completed job timestamps by partition or routing key from your metrics store"
echo "Compare rates: completed_per_sec vs publish_per_sec over the same 5-minute window from your broker metrics"
echo "Reference: https://docs.bullmq.io/guide/jobs/stalled for the official stalled-jobs definition and lockDuration contract"