What this usually means
Lag growth means the consumer is not keeping up with the producer's write rate. The textbook cause is insufficient parallelism (too few partitions or consumers), but in practice I've seen three additional classes: (1) consumer-side processing blocking — e.g., a downstream API call or database write that introduces random latency spikes; (2) broker-side throttling — the broker limits fetch responses based on fetch.max.bytes or quota violations; (3) rebalance storms — frequent group rebalances caused by session.timeout.ms being too short for garbage collection pauses, causing consumers to be kicked out and partitions to be reassigned, during which no processing happens. Each requires a different fix path.
The first ten minutes — establish facts before touching code.
- 1Run `kafka-consumer-groups --bootstrap-server BROKER:9092 --group GROUP --describe` and note the LAG column per partition; look for partitions with lag >> others
- 2Check consumer JVM heap usage and GC logs for long STW pauses: `jstat -gcutil PID 1000` and look for FGCT > 5% or concurrent mode failures
- 3Measure end-to-end latency with a custom probe timestamp in the message header; compare produce time vs consume time
- 4Enable consumer fetch manager JMX metrics: records-lag-max, records-lag-avg, and fetch-rate; a drop in fetch-rate while lag rises indicates fetch throttling
- 5Inspect broker request handler metrics: `kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Fetch` — if p99 > 100ms, the broker is overloaded
The specific files, logs, configs, and dashboards that usually own this bug.
- searchConsumer config: group.id, session.timeout.ms, max.poll.interval.ms, max.poll.records, fetch.max.bytes
- searchConsumer app logs: look for `WARN` about revoked partitions, coordinator disconnects, or rebalance timeouts
- searchBroker logs (server.log): search for `GroupCoordinator` or `Rebalance` for the consumer group
- searchJMX metrics on both consumer and broker (e.g., via Prometheus/Grafana dashboards)
- searchSystem metrics: CPU, memory, disk I/O on consumer hosts; network throughput on brokers
- searchGC logs on consumer JVM: `-Xlog:gc*:file=gc.log` — check for frequent Full GC events
Practical causes, not theory. These are the things you will actually find.
- warningmax.poll.records set too high relative to processing time — consumer exceeds max.poll.interval.ms and gets kicked out of the group
- warningDownstream system (database, external API) latency spikes cause processing to block, holding the poll() loop
- warningsession.timeout.ms too low (e.g., 10s) combined with GC pauses > session timeout — consumer fails to send heartbeats
- warningBroker fetch quota per consumer exceeded — fetch requests are throttled, reducing consumption rate
- warningPartition count mismatched: fewer consumers than partitions (by design), but a single consumer handles multiple slow partitions
- warningRebalance protocol set to EAGER (cooperative rebalancing not enabled) causing stop-the-world rebalances
Concrete fix directions. Pick the one that matches your root cause.
- buildRight-size max.poll.records and max.poll.interval.ms — e.g., set max.poll.records=500 and max.poll.interval.ms=300000 (5 min) for a 50ms/record processing time
- buildIncrease session.timeout.ms to at least 45s and heartbeat.interval.ms to 15s to tolerate GC pauses
- buildEnable cooperative rebalancing: set `partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor`
- buildImplement async processing: decouple poll loop from processing using a work queue and commit offsets after batch completion
- buildIncrease partition count if consumers are consistently at high CPU — add partitions to spread load
- buildApply per-consumer fetch quotas on broker: `kafka-configs --bootstrap-server BROKER:9092 --alter --add-config 'consumer_byte_rate=2097152' --entity-type clients --entity-name CONSUMER_ID`
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, run `kafka-consumer-groups --describe` and confirm LAG is decreasing or stable over 10 minutes
- verifiedMonitor consumer fetch-rate JMX metric: should be steady or increasing, not dropping
- verifiedCheck consumer log for rebalance events: should be zero after fix (only initial join)
- verifiedMeasure end-to-end latency with probe messages: should be back to baseline (e.g., < 100ms)
- verifiedForce a GC on consumer with `jcmd PID GC.run` and verify no rebalance occurs
Things that make this bug worse or harder to find.
- warningIncreasing max.poll.records to fix CPU underutilization — this often exacerbates lag by making processing batch larger
- warningSetting session.timeout.ms too high (e.g., 5 min) — delays failure detection and can cause partition starvation
- warningTuning only consumer configs without checking broker throttling — if broker quota is the issue, consumer changes won't help
- warningAdding more consumers without increasing partitions — extra consumers will be idle
- warningIgnoring rebalance events in logs — even silent rebalances (no errors) cause lag bursts
The Silent Rebalance Storm: How GC Pauses Caused Lag to Spike 10x
Timeline
- 09:00PagerDuty alerts: consumer lag on 'order-processing' group exceeds 100k (baseline 2k)
- 09:03Check kafka-consumer-groups: lag on partition 3 and 7 is 50k each, others <1k
- 09:05Grafana: CPU on consumer hosts at 30%, memory 60%, no obvious resource pressure
- 09:10Enable consumer JMX: fetch-rate dropped from 200/s to 50/s in last 30 minutes
- 09:15Inspect broker logs: found repeated 'Group order-processing rebalance' every 2 minutes
- 09:20Check consumer GC logs: Full GC events every 90 seconds, each lasting ~15 seconds
- 09:25Increase session.timeout.ms from 30s to 60s and heartbeat.interval.ms from 10s to 20s
- 09:30Rebalances stop — lag starts decreasing. Root cause: G1GC mixed GC pauses > 30s due to insufficient heap
- 09:45Add -XX:+UseG1GC -Xms8g -Xmx8g and tune G1HeapRegionSize to 4m. Lag back to 2k by 10:00
At 09:00, our pager fired because consumer lag on the order-processing group had climbed from a stable 2,000 to over 100,000 in 30 minutes. I checked the consumer group description and saw that only two partitions (3 and 7) had massive lag, while others were fine. That asymmetry hinted at a per-partition stall, not a uniform slowdown.
I pulled up Grafana and saw CPU at 30% and memory at 60% — no obvious resource exhaustion. But the consumer's fetch-rate metric had dropped from 200 fetches per second to 50. That's a red flag: the consumer was making fewer fetch requests even though lag was rising. I checked broker logs and found a rebalance every 2 minutes for our group. The consumer was being kicked out repeatedly.
I looked at GC logs. With `jstat -gcutil`, I saw Full GC events every 90 seconds, each taking 15-20 seconds. Our session.timeout.ms was 30 seconds. A 20-second GC pause plus network jitter caused the consumer to miss heartbeats, triggering a rebalance. During rebalance, no processing happened — lag grew. I increased session.timeout.ms to 60s and heartbeat.interval.ms to 20s, then restarted the consumer. Rebalances stopped. I also increased heap to 8GB and tuned G1GC. Lag returned to normal within 30 minutes.
Root cause
G1GC mixed GC pauses exceeding 15 seconds caused consumers to miss heartbeat intervals (session.timeout.ms=30s), triggering repeated rebalances that halted processing and allowed lag to accumulate.
The fix
Increased session.timeout.ms from 30s to 60s and heartbeat.interval.ms from 10s to 20s to tolerate GC pauses. Also increased heap from 4GB to 8GB and adjusted G1GC region size to reduce pause frequency.
The lesson
Always correlate consumer lag with rebalance frequency and GC pauses. Session timeout must be larger than maximum expected GC pause. Tune GC before throwing more partitions.
Many teams set max.poll.records to a high number (e.g., 5000) thinking it improves throughput. What actually happens: the consumer fetches a large batch and then spends significant time processing each record. If the total processing time exceeds max.poll.interval.ms (default 5 minutes), the consumer is considered dead and a rebalance occurs. During rebalance, the consumer does no processing, and lag grows.
The fix: set max.poll.records to a value that guarantees processing completes within max.poll.interval.ms. For example, if each record takes 10ms to process, max.poll.records=500 gives 5 seconds of processing, well under the default 5 minutes. Also ensure max.poll.interval.ms is large enough to cover worst-case processing time (including downstream delays). Use async processing with manual offset commits for truly unpredictable workloads.
The default rebalance protocol (Eager) stops all consumers in the group, revokes all partitions, and then reassigns them. This stop-the-world behavior causes lag spikes because no consumer processes messages during the rebalance. With many partitions, this can take seconds to minutes.
Cooperative rebalancing (StickyAssignor or CooperativeStickyAssignor) reassigns only a subset of partitions at a time, allowing consumers to continue processing the rest. To enable, set `partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor`. This reduces rebalance duration and lag impact significantly, especially in large groups.
The consumer sends heartbeats in a background thread at heartbeat.interval.ms intervals. If the broker doesn't receive a heartbeat within session.timeout.ms, it marks the consumer as dead and triggers a rebalance. Common cause: GC pauses or thread contention blocking the heartbeat thread.
I've seen teams set session.timeout.ms as low as 10 seconds to speed up failure detection. But with G1GC, even a young GC can take 100ms, and mixed GCs can take seconds. A 10-second timeout is too tight. Set session.timeout.ms to at least 45 seconds (Kafka default 45s) and heartbeat.interval.ms to one-third of that (15s). For high-throughput consumers, consider 60s/20s.
Frequently asked questions
How do I check if consumer lag is caused by rebalancing?
Look for repeated rebalance events in consumer logs (message like 'Revoking previously assigned partitions') or broker logs ('Rebalance started'). Also monitor the JMX metric `kafka.consumer:type=consumer-coordinator-metrics,name=rebalance-latency-avg`. If it spikes above 1 second, rebalances are frequent. Use a tool like Burrow to track lag per consumer group over time — sawtooth patterns indicate rebalances.
Can increasing the number of consumers always fix lag?
No. Consumers are limited by the number of partitions. If you have 10 partitions and 20 consumers, 10 will be idle. Adding consumers beyond partition count doesn't help. Also, if the bottleneck is broker throttling or downstream processing, more consumers may increase load and worsen throttling. Always check partition count and monitor broker quotas.
Why does lag grow on only a few partitions?
Uneven lag usually means a specific partition has a bad consumer. Possible causes: a rebalance assigned that partition to a slow consumer, the partition is on a hot broker (high I/O from other topics), or the partition has a key that causes processing spikes. Use `kafka-consumer-groups --describe` to see per-partition lag. Compare the consumer's processing rate per partition with `rate` metric.
What is the best way to monitor consumer lag in production?
Use a combination of: (1) JMX metrics exposed by the consumer (records-lag-max and records-lag-avg) scraped by Prometheus; (2) Burrow, a lag-checker that provides consumer group status with threshold alerts; (3) kafka-consumer-groups command for ad-hoc checks. Set alerts on lag rate of change (e.g., lag increasing > 1000 per minute) rather than absolute value.
Should I use manual or automatic offset commits?
Automatic commits (enable.auto.commit=true) are convenient but can cause data loss or duplicates. If processing takes longer than auto.commit.interval.ms (default 5s), offsets may be committed before processing completes, causing data loss on crash. For lag-sensitive apps, use manual async commits after processing each batch. This lets you control the commit frequency and ensure at-least-once semantics.