Queue partition hotspot: find uneven work distribution

The symptoms

• •One worker process or lane sustains near-100% CPU/IO utilization while peer workers processing the same queue family remain idle or low-load.
• •Queue depth, job age, or stalled-job count for a single partition climbs monotonically while other partitions stay near zero.
• •Tail latency (p95/p99) for jobs carrying a specific key, tenant id, or hash prefix is several multiples higher than the median job latency.
• •Stalled-job events increase selectively for partitions assigned to a single worker id, especially after restarts or pod rescheduling.
• •Auto-scaling adds workers but throughput does not improve because new workers never receive a meaningful share of partitions.

Likely causes

• •Key-based partitioning uses a low-cardinality key (boolean flag, small enum, tenant id cluster) so the hash space collapses onto few partitions.
• •Static partitioning or sticky assignment binds long-lived workers to the same partition until restart, and one partition's producers outpace the others.
• •Producer-side batching or fan-out introduces bursty traffic that one partition absorbs serially while others idle between bursts.
• •Worker-side rate limit, concurrency cap, or throughput quota is enforced per partition and is exceeded only by the hot partition.
• •Re-balancing delays or single-worker lock acquisition cause a recovering partition to remain parked on one node past the stall detection window.

First ten minutes

1. 01Confirm scope: capture queue name, partition strategy, worker count, and the time window in which the imbalance appeared.
2. 02Pull per-worker and per-partition metrics for the suspected queue family and rank them by utilization and queue depth.
3. 03Compare stalled-job counts per worker id and per partition to identify whether the hotspot is worker-bound or partition-bound.
4. 04Sample the partitioning key distribution from recent jobs to check cardinality and whether one key prefix dominates.
5. 05Inspect stalled-job diagnostic fields against the official stalled-job documentation to confirm the job is genuinely stalled and not just slow.
6. 06Decide whether the hotspot is a producer-side (key skew) or consumer-side (assignment skew) issue before changing any partition rule.

Evidence to collect

• •Per-partition queue depth, age of oldest job, and stalled-job count sampled at the same instant across all partitions.
• •Per-worker CPU, event-loop lag, and active-job count over the imbalance window, tagged with worker id and partition assignment.
• •Distribution histogram of the partitioning key (hash prefix, tenant id, region) for the last N jobs to quantify skew.
• •Stalled-job event payloads including job id, worker id, partition, and the stall reason recorded by the queue.
• •Recent re-balance or assignment-change events showing which worker held the hot partition across the window.

Where to look

• •Queue telemetry boundary: partition-level queue depth, stalled-job counter, and lock-holder records supplied by the queue.
• •Worker boundary: worker process metrics, concurrency settings, and the local stalled-job detection log entries.
• •Producer boundary: enqueue call site, the key used for partitioning, and any client-side batching or aggregation layer.
• •Scheduler boundary: the stalled-job stalledInterval, maxStalledCount, and lockRenewal settings that govern how stalls are detected.
• •Routing boundary: any hash-based router, theme router, or shard map that decides which partition receives a key.

Diagnostic steps

1. 01Compute the coefficient of variation or top-vs-rest ratio of per-partition depth; a large ratio indicates hotspot rather than uniform backlog.
2. 02Group stalled-job events by worker id, then by partition, to distinguish worker-bound stalls from partition-bound stalls.
3. 03Compare key distribution against worker distribution: if keys are skewed but workers are balanced, the hotspot is producer-side; if keys are uniform but workers are skewed, it is consumer-side.
4. 04Reproduce the skew with a small synthetic load using a known key distribution and measure arrival rate per partition to confirm the routing rule.
5. 05Cross-check stalled-job fields against the official stalled-job documentation to verify stall classification and avoid mistaking slow jobs for stalled jobs.
6. 06Confirm the hotspot is not a downstream dependency: measure job handler latency for hot vs cold partitions separately to rule out a slow backend.

Common mistakes

• •Adding more workers without changing the partition rule, which only increases idle capacity while the hot partition remains bound to its current holder.
• •Increasing maxStalledCount or stalledInterval to "stop the noise," which hides the hotspot and lets the hot partition grow unbounded.
• •Assuming a stalled job means the worker is dead; a stalled job can indicate the worker is healthy but throttled, while the partition is overloaded.
• •Switching the partitioning key blindly, which can break exactly-once or ordered-processing guarantees that depend on the original key.
• •Diagnosing tail latency as a backend performance issue when the latency is actually queue residence time on the hot partition.

Safe fixes

• •If evidence shows key skew: introduce a higher-cardinality key (composite key, salted hash, or per-tenant sub-key) and re-measure per-partition depth before and after.
• •If evidence shows worker-bound assignment skew: enable dynamic re-balancing or shorten the lock-renewal window so stalled partitions reassign sooner, per the stalled-job guidance.
• •If evidence shows batched producer bursts: smooth enqueue over time with client-side pacing or sub-batching so one partition does not absorb serialized bursts.
• •If evidence shows per-partition rate limit: raise the limit only on the hot partition while keeping global throughput unchanged, and document the policy change.
• •If ordering constraints prevent key change: keep the original key for ordering but add a secondary random suffix for routing, then verify ordering only within the secondary group.

Prove the fix

1. 01Per-partition queue depth and stalled-job count converge within a defined tolerance across partitions for at least one steady-state window.
2. 02Top-vs-rest ratio of per-partition depth falls below the pre-change baseline by a documented margin and stays there for the verification window.
3. 03Tail latency for jobs carrying the previously hot key drops to within the median-plus-margin band observed on cold partitions.
4. 04Adding a worker or removing a worker no longer changes the per-partition distribution shape, indicating the routing rule is no longer sticky to a single holder.
5. 05Stalled-job rate for the previously hot partition equals the average stalled-job rate of peers and does not regress after the change.

Prevention and next steps

• •Track partitioning key cardinality and skew as a first-class metric and alert when the top key exceeds an agreed share of traffic.
• •Monitor per-partition depth and stalled-job count separately from aggregate queue depth so hotspots surface before tail latency degrades.
• •Review partition strategy whenever a new tenant, region, or feature flag introduces a low-cardinality key into the routing function.
• •Test rebalancing behavior injectively by killing the hot worker and observing whether the hot partition reassigns within the stalled-job window.
• •Document the relationship between partitioning key, ordering guarantees, and per-partition limits so future changes preserve invariants.

Safe commands and checks

redis-cli -u <redis-uri> LRANGE bull:<queue>:wait <start> <end>
redis-cli -u <redis-uri> ZRANGE bull:<queue>:stalled 0 -1 WITHSCORES
redis-cli -u <redis-uri> HGETALL bull:<queue>:meta
redis-cli -u <redis-uri> KEYS 'bull:<queue>:*' | head -n 50
redis-cli -u <redis-uri> ZCARD bull:<queue>:active
redis-cli -u <redis-uri> ZCARD bull:<queue>:delayed
redis-cli -u <redis-uri> HLEN bull:<queue>:failed