What this usually means
An Erlang process mailbox grows when messages are sent to it faster than it can consume them, or when the process stops consuming altogether. In Elixir, this typically happens because a GenServer's handle_cast/2 or handle_info/2 is either blocking (e.g., on a synchronous call to a slow service), crashing (so messages pile up while the process restarts), or the mailbox holds selective receives that trap messages in the queue. Unlike a memory leak in data structures, mailbox growth is a symptom of flow imbalance or a stuck receiver. The runtime itself will not bound the mailbox; the process will keep accepting messages and the mailbox can grow to hundreds of millions of messages, consuming all memory and triggering the memory high-water mark killer.
The first ten minutes — establish facts before touching code.
- 1Run `Process.info(pid, :mailbox_size)` on suspect pids; if > 1000, you have a problem.
- 2Use `:recon.proc_count(:mailbox_size, 10)` in a live node to list top mailbox sizes.
- 3Check `:erlang.process_info(pid, :current_function)` to see what the process is currently executing – likely stuck or busy.
- 4Attach `:sys.get_state(pid)` (if GenServer) to see if state is bloating or unchanged.
- 5Use `:redbug.start('c:Pid ! _', time: 1000)` to trace messages sent to the process and identify the sender and frequency.
- 6Check for selective receives: look for `receive ... after 0` patterns in the process's code that might leave messages in the mailbox.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchProcess info: `Process.info(pid, [:mailbox_size, :current_function, :message_queue_len])`
- searchRecon: `:recon.proc_count(:mailbox_size, 10)` – top mailbox offenders across the node
- searchRedbug trace: `:redbug.start('c:Pid ! _', time: 1000, msgs: 100)` to see incoming messages
- searchGenServer code: look for blocking calls (e.g., `Task.await`, `HTTPoison.request!`) in handle_cast/2
- searchLogger or telemetry: check if logs show repeated crashes for that process (supervision restarts)
- searchObserver GUI: connect to live node and inspect process mailbox size per pid
- searchSystem memory breakdown: `:erlang.memory(:processes)` vs `:erlang.memory(:total)` to confirm process memory is dominant
Practical causes, not theory. These are the things you will actually find.
- warningGenServer handle_cast/2 makes a synchronous call (e.g., `Task.await`) that blocks, preventing handle_cast from returning
- warningSelective receive in the process traps messages that don't match the pattern, accumulating them indefinitely
- warningCaller sends messages without backpressure (e.g., `cast` from a fast loop with no throttle)
- warningProcess crashes repeatedly and the mailbox is preserved across supervisor restarts (if trapping exits)
- warningThird-party library sends messages to a process that never reads them (e.g., Phoenix PubSub or Ecto pool)
- warningA telemetry or metrics handler that accumulates events but processes them slower than the event rate
- warningGenServer {:continue, ...} callback being too slow, causing subsequent messages to queue
Concrete fix directions. Pick the one that matches your root cause.
- buildConvert synchronous blocking calls in handle_cast/2 to async (e.g., use `Task.start` or `send_after`).
- buildAdd backpressure: limit the number of messages in flight using a semaphore or a bounded queue and `handle_info(:throttle, ...)`.
- buildRemove selective receive or restructure to use `receive` with a timeout and re-queue unmatched messages.
- buildIncrease the process's mailbox capacity? No – fix the consumer. Instead, ensure handle calls never block.
- buildUse `Process.send_after/3` to batch messages or slow down producers if possible.
- buildIf the process is a GenServer, consider using `handle_call` with a timeout and handle timeouts gracefully.
- buildFor unavoidable blocking, isolate the slow work to a separate pool of processes (e.g., Task.async_stream with a max concurrency).
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter deploying fix, monitor `:recon.proc_count(:mailbox_size, 10)` – max mailbox size should drop and stabilize.
- verifiedCheck `:erlang.memory(:processes)` over an hour – trend should flatten or decrease.
- verifiedRun load test that previously caused growth; mailbox sizes should stay below a threshold (e.g., < 50).
- verifiedVerify backpressure: temporarily increase producer rate and confirm mailbox does not grow unbounded.
- verifiedUse `Process.info(pid, :mailbox_size)` periodically in production and assert it stays low.
- verifiedCheck process's `:current_function` during peak load – should not be in a blocking state.
Things that make this bug worse or harder to find.
- warningUsing `Process.info(:mailbox_size)` without checking the process is alive – may get nil and miss the problem.
- warningAssuming mailbox growth is a memory leak – it's a flow problem; reducing process memory won't help if the mailbox keeps growing.
- warningRestarting the process without draining the mailbox – messages may be lost or re-queued after restart if the process traps exits.
- warningAdding more producers before fixing the consumer – makes the problem worse.
- warningUsing `flush()` in a GenServer – dangerous because it discards messages that might be important.
- warningSetting `:message_queue_data` to `off_heap` without understanding trade-offs (can increase per-process memory but might delay GC).
The Silent Pileup: How a Blocking Cast Took Down Our Chat Service
Timeline
- 09:15PagerDuty alert: node memory > 85%. Memory graph shows steady climb over 2 hours.
- 09:18SSH into node, run `:erlang.memory(:processes)` – 3.2 GB out of 4 GB total.
- 09:20Run `:recon.proc_count(:mailbox_size, 5)` – top process has 2.1 million messages.
- 09:22Get pid of that process via recon. `Process.info(pid, :current_function)` shows `:gen_server:loop/1` – suspicious.
- 09:25Identify it as a GenServer registered as :message_dispatcher. `:sys.get_state(pid)` shows state with 50k entries.
- 09:28Read source code: handle_cast/2 calls `Redix.command!(:redix, ...)` – a synchronous call to Redis that can block.
- 09:30Run redbug trace: `:redbug.start('c::message_dispatcher ! _', time: 30, msgs: 100)` – messages arriving every 2ms, but handle_cast taking 100ms+.
- 09:35Hotfix: change `Redix.command!` to `Redix.command` and handle async reply in handle_info. Deploy to staging.
- 09:45Deploy to prod after verification. Memory stabilizes within 10 minutes.
- 09:55Run `:recon.proc_count(:mailbox_size)` – top process now has 20 messages. Memory back to 1.2 GB.
About two hours before the alert, we had deployed a new feature that added a Redis call inside a GenServer handle_cast. The cast was triggered by every chat message, which came in at ~500/sec. The Redis call was a synchronous `command!` – it blocks the GenServer until Redis responds. But Redis was under load from other services, so response times spiked to 100-200ms. The producer (LiveView process) kept casting messages, but the dispatcher could only process 5-10 per second. The mailbox grew at ~490 messages per second.
I first noticed the memory graph looked like a perfect linear ramp – that's usually not a leak but an accumulation. The recon command quickly pointed to one process with 2.1 million messages. The current_function showed it was stuck in `gen_server:loop`, which meant it was busy handling a message. The state size was 50k entries, but that alone couldn't explain 3 GB of memory. The mailbox was the culprit.
The fix was straightforward: replace the synchronous `Redix.command!` with an asynchronous version that sends a reply to the process. I used `Redix.command` and added a `handle_info` clause to process the reply. This freed the GenServer to handle more messages, and the mailbox drained within minutes. We also added a circuit breaker to slow down producers if mailbox exceeds a threshold. Lesson learned: never block in a handle_cast – if you need to call a slow service, delegate to a task.
Root cause
Blocking Redis call (Redix.command!) inside GenServer handle_cast caused the process to halt processing new messages, leading to unbounded mailbox growth.
The fix
Replaced Redix.command! with async Redix.command and handled the reply in handle_info. Added backpressure via a :telemetry event when mailbox size exceeds 1000.
The lesson
Any synchronous I/O in a GenServer handle callback is a ticking time bomb. Always use async patterns or delegate to separate processes for slow operations.
Each Erlang process has a mailbox – a FIFO queue of messages sent to it. The runtime does not impose a limit; the mailbox can grow until memory runs out. Messages are consumed only when the process executes a `receive` block that matches the message. A GenServer's handle_cast/2 is essentially a receive loop: after handling a message, it loops back to wait for the next one. If the handler blocks, the loop doesn't advance and the mailbox accumulates.
Key detail: the mailbox is stored in the process's heap, but large mailboxes can cause garbage collection to become expensive. The runtime may trigger a major GC, which pauses the process and can cause latency spikes. The mailbox itself is not counted in `:erlang.memory(:processes)`? Actually it is – each message is a term stored on the process heap. So growing mailbox directly increases process memory.
Recon's `proc_count(:mailbox_size, N)` is your first responder. It returns the N processes with the largest mailboxes. Combine with `proc_window` to see current function. For example: `:recon.proc_count(:mailbox_size, 5) |> Enum.map(fn {pid, size} -> {pid, size, Process.info(pid, :current_function)} end)`. This gives you a quick triage.
Redbug is a tracing tool for live debugging. To trace messages sent to a specific pid: `:redbug.start('c:Pid ! _', time: 10, msgs: 50)`. This will print the first 50 messages sent to Pid over 10 seconds. You can see the sender process and message content. For GenServers, you can also trace the handle_call/cast/info callbacks: `:redbug.start(['c::gen_server:handle_call', 'c::gen_server:handle_cast'], time: 5)`. This shows if the handler is being invoked at all.
The most robust pattern is to use a bounded queue (e.g., `:queue` with max size) and a separate producer process. When the queue is full, the producer either drops messages or blocks. In GenServer, you can track mailbox size via `Process.info(self(), :message_queue_len)` and if it exceeds a threshold, send a message to the producer to pause. This is coarse but effective.
Another pattern is to use `Task.async_stream` with `max_concurrency` for processing heavy work. For GenServers, consider using `handle_continue/2` to defer work after a cast, but note that `handle_continue` can also block. The safest approach: never do I/O in handle_cast/handle_info – send a message to a dedicated worker pool.
Prometheus and Telemetry can expose per-process mailbox sizes. Use `Process.info(pid, :message_queue_len)` in a periodic metric. Set an alert if any process mailbox exceeds 1000 messages for more than 1 minute. For global monitoring, use `:recon.proc_count(:mailbox_size, 1)` and expose the max mailbox size as a gauge.
In Kubernetes, you can use the BEAM's `:erlang.system_info(:process_limit)` to know max processes, but mailbox size per process is not exposed by default. Consider using `appsignal` or `newrelic` which can track process mailbox sizes. The key is to detect growth trends before memory hits a threshold.
Frequently asked questions
Does a large mailbox cause the process to use more CPU?
Yes. When the process receives a message, it must scan the mailbox to find a matching receive clause. For a selective receive, the runtime scans from the front until it finds a match. With millions of messages, this scan can be very expensive (O(n) per message). Also, garbage collection becomes heavier because the mailbox lives on the process heap, and a major GC will scan the entire mailbox.
Can I limit the mailbox size?
Erlang/OTP does not provide a built-in limit on mailbox size. You must implement your own backpressure. You can check `Process.info(self(), :message_queue_len)` in your GenServer and if it exceeds a threshold, either drop incoming messages or signal the producer to slow down. Some libraries like `Broadway` use a `:demand` protocol to control flow.
What is the difference between mailbox size and message queue length?
They are the same. `Process.info(pid, :message_queue_len)` returns the number of messages currently in the mailbox. `:mailbox_size` is an alias in some tools (like recon). Both refer to the same queue.
Will restarting the process clear the mailbox?
Yes, restarting a process clears its mailbox because the process is destroyed and recreated. However, if the process is supervised with `:temporary` or `:transient`, the mailbox is lost. If you need to preserve messages, you must drain them manually before restart. Also, if the process traps exits and is linked to other processes, those links may cause messages to be sent to the new process.
How do I detect if a process is stuck due to a selective receive?
Use `Process.info(pid, :current_function)` – if it shows `:gen_server:loop/1` or `:prim_buffer:recv_loop/1`, it's waiting for a message. But that's normal. To detect selective receive stuckness, trace the process's receive pattern. You can also check the mailbox content: `Process.info(pid, :messages)` (but this is expensive for large mailboxes). A more practical approach is to add logging in your receive clauses to see if some messages are never matched.