What this usually means
A rebalance storm occurs when consumers cannot maintain a stable membership. The root cause is almost always a mismatch between the consumer's session.timeout.ms, max.poll.interval.ms, and the actual time required to process a batch of records. The coordinator detects a consumer as dead (session timeout) or non-responsive (max poll interval exceeded) and triggers a rebalance. During the rebalance, all consumers in the group pause processing and wait for the new assignment. If the underlying cause persists, the cycle repeats indefinitely, causing a storm.
The first ten minutes — establish facts before touching code.
- 1Check broker logs for repeated 'Group ... rebalance completed' messages within seconds of each other.
- 2Run 'kafka-consumer-groups --bootstrap-server <broker> --group <group> --describe' and note the number of members changing over time.
- 3Inspect consumer logs for 'Heartbeat expired', 'Poll timeout', or 'Member ... left group' entries.
- 4Monitor the consumer's max.poll.interval.ms and session.timeout.ms configuration values.
- 5Look at the processing time of the poll loop: add a log statement before and after 'poll()' to measure duration.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchBroker logs (server.log) for rebalance events: 'Group coordinator' and 'Rebalance completed'.
- searchConsumer application logs for 'WARN' or 'ERROR' related to heartbeats or poll intervals.
- searchKafka metrics: 'kafka.consumer:type=consumer-coordinator-metrics,client-id=*' for rebalance-rate and rebalance-latency-avg.
- searchConsumer configuration file (or environment variables) for session.timeout.ms, max.poll.interval.ms, heartbeat.interval.ms.
- searchApplication profiling (CPU, GC logs) to detect long garbage collection pauses that delay poll().
- searchNetwork logs or broker connection logs for any connectivity issues causing disconnections.
Practical causes, not theory. These are the things you will actually find.
- warningmax.poll.interval.ms too low: processing a batch takes longer than the configured max interval.
- warningsession.timeout.ms too short: network hiccups or GC pauses exceed session timeout, causing coordinator to evict consumer.
- warningLarge poll records: consumer fetches many records and processing them exceeds max.poll.interval.ms.
- warningGarbage collection pauses: long GC stop-the-world events prevent heartbeats from being sent.
- warningNetwork instability: packet loss or high latency causes heartbeats to be delayed.
- warningMisconfigured heartbeat.interval.ms: set too high relative to session.timeout.ms, leading to missed heartbeats.
Concrete fix directions. Pick the one that matches your root cause.
- buildIncrease max.poll.interval.ms to accommodate the worst-case processing time of a poll batch.
- buildIncrease session.timeout.ms to tolerate brief network or GC hiccups (but keep it under request.timeout.ms).
- buildDecrease max.poll.records to limit the number of records per poll, reducing processing time.
- buildOptimize processing logic: parallelize, reduce I/O, or use asynchronous processing to shorten poll loop.
- buildTune GC: use G1GC with short pause targets, or allocate more heap to reduce GC frequency.
- buildAdd a separate heartbeat thread (Kafka 2.3+) or use cooperative rebalancing (static membership).
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter changes, monitor rebalance rate metric: it should drop to near zero under steady load.
- verifiedRun 'kafka-consumer-groups --describe' repeatedly: the member list should stabilize.
- verifiedCheck consumer lag: it should decrease steadily and not plateau.
- verifiedVerify processing throughput returns to expected baseline.
- verifiedSimulate worst-case processing by injecting a slow consumer to ensure changes hold.
Things that make this bug worse or harder to find.
- warningSetting max.poll.interval.ms too high—delays consumer health detection; keep it reasonable (e.g., 5 min).
- warningSetting session.timeout.ms too high—slows failure detection; avoid > 60s in most cases.
- warningIgnoring GC logs: long GC pauses cause rebalances even with correct timeouts.
- warningBlindly increasing timeouts without addressing processing bottlenecks—just masks the issue.
- warningUsing static group membership without understanding its implications (requires manual member IDs).
- warningAssuming all consumers have same configuration; inconsistent timeouts can cause storms too.
The 2 AM Lag Spike: A Rebalance Storm After a Schema Change
Timeline
- 02:05PagerDuty alert: consumer group 'order-processor' lag > 500k messages
- 02:07Checked Kafka broker logs: repeated 'Rebalance completed' every 30 seconds
- 02:10Ran kafka-consumer-groups --describe; group had 16 members, changing every minute
- 02:12Checked consumer logs: found 'Max poll interval exceeded' warnings
- 02:15Reviewed recent changes: team had added a new Avro schema validation that introduced 200ms latency per message
- 02:20Calculated: 500 records per poll * 200ms = 100s processing time, but max.poll.interval.ms was 60000 (60s)
- 02:22Increased max.poll.interval.ms to 180000 (3 min) and reduced max.poll.records to 200
- 02:25Rebalances stopped; lag started dropping
- 02:40Lag back to normal; team created a JIRA to optimize schema validation
I was on-call when the pager went off at 2 AM. The 'order-processor' consumer group had blown past 500k lag and was still climbing. My first instinct was to check if a broker had died, but the cluster was green. Then I looked at the broker logs and saw the telltale sign: 'Rebalance completed' messages repeating every 30 seconds. A rebalance storm.
I jumped into the consumer logs and found 'Max poll interval exceeded' warnings. That's the smoking gun: the consumer was spending more time processing than the allowed max.poll.interval.ms. I checked recent deployments and saw the team had added Avro schema validation using a library that made a network call per message. That increased per-record processing time from 2ms to 200ms.
With 500 records per poll, that was 100 seconds of processing. But max.poll.interval.ms was set to 60 seconds. The fix was straightforward: increase max.poll.interval.ms to 180 seconds and reduce max.poll.records to 200 to limit the processing time. I applied the change, the rebalances stopped, and the lag started draining. We later optimized the schema validation to cache results, but the immediate fix saved the night.
Root cause
A code change introduced a high-latency operation (Avro schema validation) per message, causing poll processing time to exceed max.poll.interval.ms.
The fix
Increased max.poll.interval.ms from 60s to 180s and reduced max.poll.records from 500 to 200 to keep processing under the new interval.
The lesson
Always monitor the actual poll loop duration and set max.poll.interval.ms with headroom. Any change affecting per-message latency can trigger a rebalance storm.
Kafka's consumer rebalance protocol is cooperative: the group coordinator elects a leader who computes partition assignments and distributes them. During a rebalance, all consumers pause processing. A storm happens when consumers repeatedly trigger rebalances because they appear dead or non-responsive.
The coordinator uses two mechanisms to detect failures: session.timeout.ms (heartbeat missed) and max.poll.interval.ms (no poll() call). If any consumer fails either check, the coordinator initiates a rebalance. The storm continues until the underlying cause is fixed.
session.timeout.ms: The maximum time between heartbeats. Default 45s. If a heartbeat is not received, the consumer is considered dead and removed from the group. Set this high enough to handle transient network or GC pauses, but low enough for fast failure detection.
max.poll.interval.ms: The maximum time between poll() calls. Default 5 minutes. If the processing loop takes longer, the consumer is considered failed. This is the most common culprit in storms. Set it to a value that covers the worst-case processing time of a poll batch.
heartbeat.interval.ms: The frequency of heartbeats. Default 3s. Should be set to roughly one-third of session.timeout.ms to ensure timely heartbeats. If set too high, the coordinator may time out before receiving a heartbeat.
Kafka 2.3+ introduced static group membership, where consumers have a fixed member ID (via group.instance.id). This allows the coordinator to keep partition assignments during a restart, preventing a full rebalance. However, it introduces complexity: the coordinator waits for the member to return within session.timeout.ms.
Static membership is useful for planned restarts but doesn't fix storms caused by processing timeouts. It can mask the symptom but the underlying issue (slow processing) still exists. Use it as a complement to proper timeouts, not a replacement.
Key metrics to monitor: kafka.consumer:type=consumer-coordinator-metrics,client-id=* attributes: rebalance-rate (events per second), rebalance-latency-avg (average time per rebalance), and rebalance-latency-max. Set alerts when rebalance-rate exceeds a baseline (e.g., >0.1 per second).
Also monitor the consumer's max.poll.interval.ms exceeded count via JMX metrics (if available) or by logging warnings. Lag metrics from kafka-consumer-groups can indicate a storm when lag increases despite no broker issues.
Use kafka-consumer-groups --bootstrap-server <broker> --group <group> --describe --state to get detailed group state. A rebalance storm will show the group in 'PreparingRebalance' or 'CompletingRebalance' state frequently. Check the 'members' list for any member with 'UNKNOWN' status.
Enable TRACE logging for the consumer's coordinator class (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) to see heartbeat failures and rebalance triggers. This can pinpoint whether it's a session timeout or max poll interval issue.
Frequently asked questions
What's the difference between a rebalance storm and a slow consumer?
A slow consumer causes lag to increase but does not necessarily trigger rebalances. A rebalance storm involves repeated rebalances that stop processing entirely, whereas a slow consumer continues processing but at a reduced rate. Both cause lag, but the storm is more disruptive because all consumers pause during rebalances.
Can a rebalance storm be caused by a broker issue?
Yes, if the group coordinator broker is overloaded or crashes, it can cause frequent rebalances. However, the most common cause is consumer-side misconfiguration (timeouts or processing delays). Check broker logs for coordinator failures and broker CPU/memory.
How do I calculate the right max.poll.interval.ms value?
Profile the actual time taken to process a poll batch under worst-case conditions. Add a 50% buffer. For example, if processing 500 records takes 60 seconds on average but can spike to 90 seconds, set max.poll.interval.ms to 180 seconds (3x the spike). Also consider GC pauses.
Should I disable rebalances entirely?
No, rebalances are essential for scaling and fault tolerance. Instead, use static group membership to reduce unnecessary rebalances during planned restarts, and ensure timeouts are properly configured. Disabling rebalances would require manual partition assignment, which is not scalable.
What if increasing timeouts doesn't stop the storm?
If timeouts are already high and storms persist, check for network issues, GC pauses, or thread contention. Use thread dumps to see if the poll loop is blocked on I/O or locks. Also verify that all consumers in the group have the same timeout configurations.