Debugging Performance Backpressure

How this failure announces itself.

• warningA worker's heap grows steadily under load until the process is OOM-killed.
• warningMetric series counts or queue depths spike together with traffic.
• warningOne tenant's burst starves every other tenant of a shared resource.

The first ten minutes — establish facts before touching code.

• 1Capture a heap profile or memory timeline during load and identify what is accumulating.
• 2Find the producer of that accumulation and check whether anything ever tells it to slow down.
• 3Measure consumer throughput against producer rate to size the gap.
• 4Check limiter scope against the failure: who was supposed to be limited and was not?

The code and config that usually owns this bug.

• searchStream write return values - false means stop until drain, and ignoring it buffers without bound.
• searchDrain handling - does the producer actually pause, or just keep pushing?
• searchLimiter scopes - a global limiter lets one tenant consume the whole budget; a per-call limiter limits nothing.
• searchTelemetry dimensions - backpressure problems often surface first as cardinality growth.

Fix the cause, then make the regression impossible.

• buildRespect drain: pause the producer when write returns false and resume on the drain event.
• buildBound every internal buffer and decide explicitly what happens at the bound - block, shed, or sample.
• buildScope shared resources per tenant or key so one consumer cannot starve the rest.
• buildAdd a peak-memory assertion to load tests so regressions fail before production.

A fix you can't demonstrate is a guess. Close the loop.

• verifiedA load test above consumer capacity shows memory plateauing instead of climbing.
• verifiedThe slow-consumer test shows the producer pausing on drain and resuming after.
• verifiedOne tenant's burst no longer degrades another tenant's latency in tests.