Queue starvation: distinguish no work from unreachable work

The symptoms

• •Specific partitions or priority tiers process at full rate while others sit idle; aggregate throughput stays green
• •Per-partition lag grows unbounded on a subset of partitions while overall consumer lag looks healthy
• •Worker pool shows lopsided assignment: some consumers fully busy, others idle with no partitions assigned
• •High watermark on starved partitions advances (producers are publishing) but consumer offsets do not move
• •Alerting fires on backlog growth for specific keys or tiers without a corresponding drop in total throughput

Likely causes

• •Consumer rebalance left one or more partitions unassigned after a worker restart, scale event, or session timeout
• •Static or pinned partition assignment that excluded newly created partitions or specific keys
• •Priority queue weight or preemption settings that let lower-priority work dominate capacity under load
• •Cooperative-sticky rebalance behavior in which a partition was orphaned and never picked up by another member
• •Network, DNS, or broker-routing failure that isolates a consumer from a subset of partitions while others remain reachable
• •Auth, ACL, or TLS change that permits consumption of some topics but denies the affected partition host

First ten minutes

1. 01Capture the issue window: timestamp of first observed backlog growth and last successful processing per affected partition or tier
2. 02List which partitions, keys, or priority tiers are starved versus which are flowing normally
3. 03Record current consumer count, registered group members, and the partition-to-member assignment snapshot
4. 04Look for any deploy, scale, config change, or broker failover that aligns with the onset window
5. 05Freeze state: snapshot the assignment, offsets, and logs before triggering any rebalance or restart

Evidence to collect

• •Per-partition consumer offsets and lag at the moment starvation began and at the current observation point
• •Consumer group member list, generation id, and full partition-to-member assignment map
• •Rebalance and group-coordinator log entries with timestamps for the affected group
• •Topic high-watermark progression on the starved partitions to confirm producers are publishing
• •Recent deployment, scaling, configuration, ACL, or DNS change log correlated with the onset window

Where to look

• •Consumer group describe output showing which member owns each partition and whether any partition has no owner
• •Group coordinator logs filtered for rebalance start, sync group, and assignment completion events
• •Topic-level offset and lag metrics broken down per partition, not aggregated
• •Worker process logs filtered for assignment change, heartbeat failure, or fetch error entries
• •Broker logs for the affected partitions to rule out leader-change storms or throttling

Diagnostic steps

1. 01Confirm the symptom shape: plot per-partition lag and per-partition consumption rate. Starvation manifests as near-zero rate on a specific subset while the rest of the system is healthy — not as a uniform slowdown.
2. 02Enumerate the assignment: list every partition and its owning consumer. If one or more partitions have no owner, the problem is assignment, not absence of work.
3. 03Distinguish "no work" from "unreachable work": verify the high watermark is advancing on the starved partitions and that earliest-to-latest offset deltas are non-zero. If offsets move upstream but downstream is flat, work exists and is not being reached.
4. 04Trace the last rebalance: identify the rebalance that produced the current generation. If a partition dropped out of assignment at that point and no subsequent rebalance reclaimed it, suspect assignment loss, not a worker bug.
5. 05Inspect consumer-side blockers on the would-be owners: long GC pauses, blocked fetches, thread starvation, or authentication failures can prevent a consumer from claiming its assigned partitions even when assignment appears correct.
6. 06For priority queues, reproduce in a controlled load test that isolates the priority mechanism; if low-priority work preempts high-priority work as designed, the issue is policy, not infrastructure.
7. 07Cross-check routing or pinning logic: if a hash, key range, or partitioner pins certain keys to specific partitions that no longer have an owner, starvation follows from the routing rule, not from capacity.

Common mistakes

• •Restarting consumers immediately, which can mask an assignment problem and destroy the evidence needed to diagnose it
• •Scaling workers up without verifying whether partitions were actually reassigned to the new members
• •Concluding "no messages exist" because a UI shows zero, without checking earliest/latest offset and high watermark movement
• •Treating priority starvation as a worker bug when the configured policy is the cause
• •Deleting or replaying messages without first verifying they were actually delivered to a consumer

Safe fixes

• •Trigger a scoped rebalance by restarting one consumer at a time and observing whether orphaned partitions are reclaimed; do not restart the whole group at once
• •If assignment is pinned or skewed by configuration, adjust the partition assignment strategy or partitioner so the affected keys map to partitions with active owners
• •For priority queues, adjust weight or preemption parameters via a configuration change and re-observe distribution under controlled load before promoting the change
• •If the onset correlates with a specific change, roll back that change first and confirm whether starvation resolves before attempting structural fixes
• •Add per-partition lag and assignment-coverage metrics before scaling further, so future starvation is detectable without aggregate-only dashboards

Prove the fix

1. 01Starved partitions resume non-zero consumption rate within minutes of the fix and remain stable through at least one further rebalance
2. 02Per-partition lag for the previously starved partitions begins decreasing and converges toward lag on healthy partitions
3. 03Consumer group assignment map shows full coverage with no partition left without an owner across a full rebalance cycle
4. 04High watermark continues to advance and consumer offsets track it on every partition, confirming end-to-end reachability
5. 05For priority queues, a controlled load test reproduces the expected distribution without high-priority work being indefinitely deferred

Prevention and next steps

• •Treat per-partition lag and assignment coverage as first-class signals; never rely solely on aggregate throughput or aggregate lag dashboards
• •Make rebalance events a deployment-relevant signal: log generation id, member set, and resulting assignment, and review them during change windows
• •Test priority preemption and partition pinning behavior in staging under realistic load, including scale-out and scale-in scenarios
• •Alert on any partition that has a non-zero high watermark but no active consumer owner for longer than a defined threshold

Safe commands and checks

kafka-consumer-groups.sh --bootstrap-server <bootstrap-server> --describe --group <group-id>  # read-only: list partitions, consumer members, current offsets, and lag for the group. Obtain <bootstrap-server> from your cluster config and <group-id> from the client configuration.
kafka-consumer-groups.sh --bootstrap-server <bootstrap-server> --describe --all-groups  # read-only: enumerate all groups and their state to spot groups stuck in rebalance or with no active members.
kafka-run-class.sh kafka.tools.GetOffsetShell --bootstrap-server <bootstrap-server> --topic <topic-name>  # read-only: read earliest and latest offsets per partition to confirm whether the high watermark is advancing on starved partitions.
kafka-topics.sh --bootstrap-server <bootstrap-server> --describe --topic <topic-name>  # read-only: confirm partition count, replication factor, and leader assignment for the affected topic.
jstack <pid> | sed -n '/^"main"/,/^$/p'  # read-only: dump JVM thread state for a suspect consumer. Obtain <pid> from `ps -ef | grep <consumer-process-name>` on the worker host; never kill or modify the process from this command.
grep -E 'rebalance|AssignedPartitions|SyncGroup|Heartbeat' <consumer-log-path>  # read-only: extract rebalance-related entries from the consumer log to find the generation where a partition was orphaned. Replace <consumer-log-path> with the actual log location.