Backpressure in Streaming Systems Explained

I've seen the same pattern play out at three different companies: a team builds a streaming pipeline, it works fine in staging, then on production with real traffic, the consumer falls behind, memory usage climbs, and eventually the JVM crashes with an OutOfMemoryError. The root cause is almost always the same: no backpressure.

Backpressure is the system's way of telling the producer "slow down, I can't keep up." It's a fundamental concept in stream processing, but it's often treated as an afterthought. In this post, I'll explain what backpressure looks like in practice, how Kafka and Flink handle it, and how to build backpressure-aware pipelines that don't fall over under load.

The Silent OOM Incident

14:00Deploy new streaming pipeline consuming from Kafka topic with 50 partitions.
14:05Consumer group lag starts rising. Consumer logs show frequent GC pauses.
14:07First consumer node hits 'java.lang.OutOfMemoryError: Java heap space' and crashes.
14:08Rebalancing triggers. Remaining consumers get more partitions, worsening load.
14:10Second consumer OOM. Cascade begins.
14:12All consumers dead. Pipeline down. No alert on lag because team only monitored processing rate, not memory.

Lesson

The pipeline had no backpressure mechanism. The consumer polled as fast as Kafka would deliver, filling up an in-memory buffer that was never bounded. A simple backpressure strategy (e.g., limiting poll size or using Reactive Streams) would have prevented the crash by capping memory usage.

What Backpressure Actually Means

In a streaming system, data flows from producers through one or more stages to consumers. Each stage has a finite processing capacity (e.g., records/second). If a downstream stage is slower than the upstream one, data piles up. Without backpressure, that pile grows unbounded, consuming memory until the system fails.

Backpressure flips the flow: the consumer tells the producer how much data it can handle. This is often implemented via a 'request(n)' protocol where the consumer requests N items, and the producer sends at most N. The producer then waits until it receives another request.

info

Backpressure is not the same as congestion control. Congestion control (e.g., TCP) is about network capacity; backpressure is about application-level processing capacity.

Kafka's Implicit Backpressure

Kafka consumers poll for records in a loop. The poll() call returns a batch of records. By default, the consumer can fetch up to 500 records per poll (max.poll.records). This acts as a crude backpressure: the consumer only fetches as many records as configured. If processing is slow, the consumer can reduce max.poll.records to limit memory usage.

But there's a catch: the consumer must call poll() within max.poll.interval.ms (default 5 minutes) to avoid being considered dead. If processing a batch takes longer, you need to increase that interval or use manual offset commits. Many teams set max.poll.records too high, thinking it improves throughput — it actually increases memory pressure and latency.

Simple Kafka consumer with max.poll.records set to 100. This limits the number of records fetched per poll, providing backpressure.

// Kafka consumer with explicit backpressure via max.poll.records
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("enable.auto.commit", "false");
props.put("max.poll.records", "100"); // limit batch size

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("my-topic"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        process(record); // may be slow
    }
    consumer.commitSync();
}

Reactive Streams and the request(n) Protocol

For more dynamic backpressure, the Reactive Streams specification (Java 9 Flow API) defines a standard: a Publisher sends data to a Subscriber, and the Subscriber controls flow via Subscription.request(n). The Publisher must never send more than the requested amount. This is the gold standard for backpressure in JVM-based streaming.

Libraries like RxJava, Project Reactor, and Akka Streams implement this. Flink uses a similar credit-based mechanism internally.

Reactive Streams subscriber that controls backpressure by requesting 10 items at a time.

// Reactive Streams example using Java 9 Flow API
SubmissionPublisher<Integer> publisher = new SubmissionPublisher<>();

publisher.subscribe(new Flow.Subscriber<>() {
    private Flow.Subscription subscription;
    private int requested = 0;

    @Override
    public void onSubscribe(Flow.Subscription subscription) {
        this.subscription = subscription;
        subscription.request(10); // ask for 10 items
    }

    @Override
    public void onNext(Integer item) {
        process(item);
        requested++;
        if (requested == 10) {
            subscription.request(10); // request more as we finish
            requested = 0;
        }
    }

    @Override
    public void onError(Throwable throwable) { }

    @Override
    public void onComplete() { }
});

Flink's Credit-Based Flow Control

Apache Flink implements a sophisticated credit-based flow control between operators. Each downstream task has a buffer pool. It grants a number of 'credits' (buffers) to the upstream task. The upstream can only send as many records as the number of credits it has. This prevents head-of-line blocking and memory buildup.

Flink's approach is more efficient than TCP backpressure because it works at the application layer and can prioritize data shuffles. In practice, you can tune buffer sizes (taskmanager.memory.segment-size, taskmanager.network.memory.buffers-per-channel) to adjust backpressure behavior.

Throughput improvement seen after tuning Flink network buffers from default 2 MB to 8 MB per channel in a high-shuffle workload.

Designing Backpressure into Your System

1Measure your pipeline's bottleneck: monitor consumer lag, CPU, memory, and GC. Use tools like Burrow (for Kafka) or Flink's web dashboard.
2Set explicit limits: Kafka max.poll.records, Flink buffer sizes, or Reactive Streams request(n).
3Implement circuit breakers: if a downstream service is slow, fail fast rather than queue indefinitely. For example, use Hystrix or resilience4j.
4Use bounded queues: if you must buffer, use an ArrayBlockingQueue with a limit. Never use LinkedBlockingDeque without a capacity.
5Test under load: simulate peak traffic and observe backpressure behavior. Tools like Gatling or custom Kafka producer scripts can help.

A pipeline without backpressure is like a firehose aimed at a teacup. It's not a question of if it overflows, but when.

When Backpressure Fails

Backpressure is not a silver bullet. If the consumer is permanently slower than the producer, backpressure will eventually cause the producer to block, which can cascade upstream. This is why you also need elastic scaling (auto-scale consumers) and dead-letter queues for records that cannot be processed.

Another anti-pattern is ignoring backpressure signals. I've seen teams set max.poll.records to Integer.MAX_VALUE thinking it's 'optimized.' It's not — it's a memory bomb. Always respect the feedback.

Batch Systems and Backpressure

Even batch systems benefit from backpressure. Spark Streaming uses a 'rate controller' that estimates processing time and adjusts the batch interval. If the system is falling behind, it increases the interval to reduce load. This is essentially backpressure at the batch level.

Enabling Spark Streaming backpressure via configuration.

// Spark Streaming backpressure: enable rate controller
spark.conf.set("spark.streaming.backpressure.enabled", "true")
spark.conf.set("spark.streaming.backpressure.initialRate", "1000")

val stream = SparkSession.builder()
  .appName("BackpressureExample")
  .getOrCreate()
  .readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "localhost:9092")
  .option("subscribe", "my-topic")
  .load()
  .selectExpr("CAST(value AS STRING)")
  .writeStream
  .format("console")
  .option("truncate", "false")
  .start()

Wrapping Up

Backpressure is not a nice-to-have — it's a core design requirement for any streaming system that expects to handle variable load. Whether you use Kafka's max.poll.records, Reactive Streams' request(n), or Flink's credit-based flow control, the principle is the same: let the consumer dictate the pace.

The next time you see OOM in a streaming pipeline, don't just increase memory. Ask: where is the backpressure? If the answer is 'nowhere,' you've found the real bug.

Frequently asked questions

What is backpressure in streaming systems?

Backpressure is a feedback mechanism that lets a downstream consumer signal an upstream producer to slow down when it cannot keep up with the data rate. It prevents buffer overflow, memory exhaustion, and data loss.

How does Kafka implement backpressure?

Kafka consumers poll for records in a loop. By setting max.poll.records to a low value, the consumer effectively backpressures itself: it only fetches as many records as it can process within max.poll.interval.ms. The broker buffers the rest.

What is credit-based flow control in Flink?

Flink uses a credit-based mechanism where each downstream task grants a number of 'credits' to upstream tasks. An upstream task can only send as many buffers as the number of credits received. This avoids memory buildup at the receiver.

Can backpressure cause deadlocks?

Yes, if not designed carefully. For example, if a consumer holds a lock while waiting for a slow downstream, and the producer needs that lock to release data, a circular dependency can cause a deadlock. Reactive Streams' request(n) model avoids this by decoupling demand.

Backpressure in Streaming Systems: When Your Pipeline Fights Back