What this usually means
The underlying cause is almost always a mismatch between the producer's write speed and the consumer's drain rate. Node.js streams use an internal buffer (highWaterMark) that, when full, signals the producer to stop writing via `write()` returning `false`. But many developers ignore that return value or never listen for the `'drain'` event. In complex pipelines, a slow consumer (e.g., a gzip compressor writing to a slow disk) fills the buffer, the producer keeps pushing, and either memory balloons or writes silently fail. The default highWaterMark for objectMode is 16 — far too small for large records, leading to constant backpressure. For buffers, 16KB is the default, which can be too large or too small depending on the I/O pattern.
The first ten minutes — establish facts before touching code.
- 1`strace -e trace=write,read -p <pid>` to see actual system call sizes — if writes stall, the kernel buffer is full.
- 2`node -e "const s = require('stream').PassThrough({highWaterMark: 16384}); s.write(Buffer.alloc(16384)); console.log(s.write(Buffer.alloc(1)));"` — returns false when buffer exceeds highWaterMark.
- 3Add a `'drain'` event listener on the writable stream and log timestamps to see if it fires at all.
- 4Monitor heap usage: `process.memoryUsage().heapUsed` before and after a large write — if it spikes, backpressure isn't working.
- 5Check `stream.writableLength` property in debug logs — if it exceeds highWaterMark, writes are being buffered.
- 6Use `async` iterators with `for await...of` — they handle backpressure correctly by awaiting `next()` after each chunk.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchStream pipeline code: every `write()` call must check return value and wait for `'drain'`.
- search`process.memoryUsage().heapUsed` before and after stream processing in production logs.
- searchKernel write buffer stats: `cat /proc/$(pgrep -f node)/fdinfo/1` for file descriptor write position.
- searchNode.js stream state: `stream._writableState.buffered.length` in debug mode.
- searchApplication-level metrics: if using `pino` or `winston`, log `stream.writableLength` periodically.
- search`highWaterMark` value in stream constructor — defaults: 16KB for buffer, 16 for objectMode.
- searchConsumer slow path: check if the consuming end has a backpressure mechanism of its own (e.g., database connection pool, HTTP response).
Practical causes, not theory. These are the things you will actually find.
- warningIgnoring `write()` return value: code never checks for `false` and never listens for `'drain'`.
- warningDefault highWaterMark too small for object streams (16 objects) causing constant backpressure thrashing.
- warningConsumer slower than producer with no buffering in between (e.g., writing to a network socket with small send buffer).
- warningUsing `pipe()` without handling the `'error'` event — if destination errors, source keeps writing.
- warningMultiple writers to the same stream without synchronization — one writer fills buffer and another never gets drain.
- warningUsing `Transform` with oversized `writableHighWaterMark` but tiny `readableHighWaterMark`, causing internal memory imbalance.
- warningStreams in `objectMode` with large objects: internal buffer count limited (16) but each object might be huge, leading to memory pressure.
Concrete fix directions. Pick the one that matches your root cause.
- buildCheck `write()` return: if `false`, pause the source or await a Promise that resolves on `'drain'`.
- buildImplement a manual buffering queue: push data to an array when `write()` returns `false`, then drain it on `'drain'`.
- buildTune `highWaterMark` based on expected object size: for large objects, set to 1 or 2; for high throughput, increase to 65536+.
- buildUse `stream.pipeline()` with a `finished` callback — it automatically handles backpressure and errors.
- buildConvert to async generator with `Readable.from()` and `for await...of` — backpressure is built-in via `next()` waiting for `'drain'`.
- buildAdd a `Transform` that throttles based on `writableLength` — if it exceeds a threshold, delay the `push()`.
- buildFor object streams, batch objects into arrays to reduce the number of writes and avoid the 16-object limit.
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, stream same dataset and compare memory usage: `node -e "..." | grep heapUsed` — should be flat.
- verifiedCount records: `wc -l input.txt` vs `wc -l output.txt` — should match.
- verifiedCheck that `'drain'` event fires periodically: add a counter in the listener and log it every 100 events.
- verifiedRun under load test: `artillery quick -n 1000 -r 10` while piping through the stream — no 'write after end' errors.
- verifiedMonitor `writableLength`: should never exceed highWaterMark for more than a few milliseconds.
- verifiedUse `--trace-warnings` and check for `(node) warning: possible EventEmitter memory leak detected` — often from unbounded drain listeners.
Things that make this bug worse or harder to find.
- warningNever ignore `write()` return value — it's the only signal for backpressure.
- warningDon't set `highWaterMark` to `Infinity` — it bypasses backpressure entirely and will crash the process on large data.
- warningAvoid using `pipe()` in production without error handlers — it swallows errors from the destination.
- warningDon't assume `'drain'` will fire after a single `write(false)` — it fires only when buffer drops below highWaterMark.
- warningNever use `process.nextTick()` to resume writing — it can cause starvation; use `setImmediate()` or a proper drain handler.
- warningDon't forget to handle `'error'` on all streams in a pipeline — unhandled errors will crash the process.
Silent Data Loss in CSV Transformation Pipeline
Timeline
- 09:15Deploy new CSV transformation service to staging.
- 09:47QA reports output files are missing the last 10-20% of rows.
- 10:02Check logs: no errors, pipeline completes in ~30s.
- 10:15Add debug logging: `write()` returns `false` after ~15,000 records.
- 10:22Realize drain event never fires after first false.
- 10:30Check highWaterMark: default 16 (objectMode) for Transform stream.
- 10:35Fix: increase highWaterMark to 1000 and add drain handling.
- 10:42Rerun pipeline: all 50,000 rows present, memory flat.
- 11:00Deploy fix to production.
We had a pipeline: CSV file from S3 -> parse rows -> transform (add timestamp) -> gzip -> upload to S3. The gzip stream (zlib.createGzip()) was the bottleneck. On my machine with fast SSD, it worked fine. In staging with slower EBS, the gzip stream's internal buffer filled quickly. The Transform stream between parser and gzip had default highWaterMark of 16 objects. After 16 objects queued, `write()` returned `false`. But I had never checked it — I just kept calling `write()` naively. Those writes were silently buffered in the Transform's internal buffer, which grew unbounded. The parser finished all rows, the Transform emitted 'finish', and the pipeline closed before the gzip had flushed everything. The last 8,000 rows were stuck in the Transform's buffer and never reached the output.
When I added logging, I saw `writableLength` shoot from 16 to 50,000+ objects. The `'drain'` event never fired because I never paused the source. Node.js will not emit `'drain'` unless `write()` previously returned `false` and the stream is waiting for drain — but I never stopped writing, so it never needed to drain. The memory grew to 1.2GB before the pipeline finished, and the last chunk of data was orphaned.
I fixed it by checking `write()` return value. When it returned `false`, I paused the parser (using `pause()` on the read stream from S3) and waited for `'drain'` before resuming. I also increased the Transform's `writableHighWaterMark` to 1000 objects, which reduced the frequency of backpressure cycles. After that, memory stayed under 200MB and all rows appeared in the output. The lesson: never trust that `write()` will always succeed; backpressure is the contract, and ignoring it is data loss.
Root cause
Ignoring `write()` return value and missing `'drain'` event listener caused the Transform stream to buffer an unlimited number of objects, leading to memory exhaustion and data loss on pipeline end.
The fix
Check `write()` return value; if `false`, pause the source stream and wait for `'drain'` before resuming. Also increased `writableHighWaterMark` to 1000 to reduce backpressure thrashing.
The lesson
Always honor the backpressure signal: if `write()` returns `false`, you must stop writing until `'drain'` fires. Default highWaterMark is too small for many production workloads.
Node.js streams are built on an internal state machine defined in `lib/_stream_writable.js`. When `write()` is called, the data is added to the internal buffer (`_writableState.buffered`). If the buffer length exceeds `highWaterMark`, `write()` returns `false`. This is the backpressure signal. The stream will emit `'drain'` once the buffer has been drained below `highWaterMark` again. The key insight: `'drain'` only fires if the stream was in a 'writing' state and has now become writable again. If you never stop writing, `'drain'` is never emitted because the stream never transitions from 'full' to 'not full' — it's constantly full.
The default `highWaterMark` for objectMode is 16, meaning the stream will buffer up to 16 objects before signaling backpressure. This is fine for small objects, but if each object is a 10MB CSV row (unlikely but possible), memory balloons to 160MB before backpressure kicks in. For Buffer mode, the default is 16KB. In either case, tuning this value is critical. A common pattern is to set `highWaterMark` to `os.freemem() / 100` for large buffers, but for objectMode, consider the expected object size.
When using `pipe()`, the source automatically handles backpressure — it pauses when destination `write()` returns `false` and resumes on `'drain'`. However, `pipe()` does not forward errors, so a failure in the destination can cause the source to keep writing indefinitely. Use `pipeline()` from the `stream` module instead, which handles errors and cleanup.
The first sign is often memory growth. Use `process.memoryUsage().heapUsed` and log it every 1000 writes. If heap grows linearly with data size, backpressure is not working. Another indicator is the `writableLength` property of a `Writable` stream — it shows the number of bytes/objects buffered. Log this after each `write()`. If it exceeds `highWaterMark` consistently, you have a backpressure problem.
For pipelines, add a `Transform` that counts chunks and logs a warning if `writableLength > highWaterMark * 2`. This gives you an early alert. Also monitor the `'drain'` event frequency — if it never fires, either your source is too slow (unlikely) or you never pause. Finally, check system-level write buffers: `strace -e trace=write -p <pid>` shows the size of each `write()` syscall. If you see many small writes (e.g., 16KB), the kernel buffer may be full.
Modern Node.js supports async iterators, which handle backpressure naturally. For example, `for await (const chunk of readable)` will automatically wait for the next chunk to be available, and the underlying stream will manage backpressure. You can also create a `Writable` stream that uses a promise-based approach:
``` class BackpressureWriter extends Writable { _write(chunk, encoding, callback) { // async write someAsyncWrite(chunk).then(() => callback()).catch(callback); } } ```
This works because `callback()` signals that the chunk has been processed. If you delay calling `callback()`, the internal buffer will fill and eventually backpressure the source. For complex pipelines, consider using `Readable.from()` with an async generator that yields chunks only when the consumer is ready.
Another pattern is to use `highWaterMark` as a warning threshold, not a hard limit. Set a custom `'drain'` handler that also throttles the producer via a semaphore or external queue. This gives you more control than the built-in mechanism.
Some developers set `highWaterMark: 1` to force backpressure after every write. This can work but introduces latency because the stream emits `'drain'` after every chunk. The overhead of context switching may reduce throughput. It's better to set a reasonable `highWaterMark` (e.g., 1000 for objects) and let the stream batch naturally.
However, if you have a very fast producer and a very slow consumer, `highWaterMark: 1` might be appropriate to limit memory. In that case, ensure you handle `'drain'` correctly and avoid writing multiple chunks before drain fires. A common mistake is to write many chunks synchronously after a single `drain` event — this defeats the purpose. Always write one chunk per `'drain'` cycle, or use a queue.
Frequently asked questions
What exactly is highWaterMark and how do I choose the right value?
highWaterMark is the internal buffer size threshold (in bytes for Buffer mode, count for objectMode) at which `write()` returns `false`. The default is 16KB for buffers and 16 for objects. Choose a value that balances memory usage and throughput. For high-throughput binary streams, 64KB to 1MB is common. For objectMode, estimate the average object size and set highWaterMark so that total buffered memory doesn't exceed a fraction of available RAM. For example, if each object is 1MB, a highWaterMark of 2 means at most 2MB buffered.
Does `pipe()` handle backpressure automatically?
Yes, `pipe()` pauses the source when destination `write()` returns `false` and resumes on `'drain'`. However, `pipe()` does not forward errors from the destination to the source. If the destination errors, the source may keep writing and cause memory issues. Use `pipeline()` from the `stream` module instead, which handles errors and cleanup properly.
How can I test if my stream handles backpressure correctly?
Create a mock consumer that is slow—use a `Writable` that delays `callback()` by 100ms. Pipe a large dataset through it and monitor memory usage. If memory grows unbounded, your producer isn't pausing. Also check that `'drain'` events fire. You can also use the built-in `stream.finished()` to ensure the pipeline completes correctly.
What happens if I continue writing after `write()` returns false?
Node.js will buffer the data internally, but the buffer size is not limited. Eventually, the process will run out of memory and crash. Additionally, if the stream ends (e.g., `end()` is called) while data is still buffered, that data is lost. The `'drain'` event will never fire because the stream is never given a chance to drain. Always respect the `false` return.
Can backpressure cause deadlocks?
Yes, if two streams are piped in a cycle (e.g., A -> B -> A) or if two streams wait on each other's `'drain'` events without writing. In practice, this is rare but possible with custom `Transform` streams that both read and write. To avoid deadlocks, ensure that at least one stream in a cycle can buffer data, or use a deadlock detection mechanism like timeouts.