What this usually means
A GenServer call timeout means the server received your message but didn't reply within the timeout window. Since GenServers handle messages sequentially in a single process, the most common cause is a synchronous call (or cast) that blocks the server's receive loop for longer than expected. Typical blockers include :timer.sleep/1, GenServer.call/3 to another slow process (nested calls), synchronous HTTP requests without a timeout, database queries that hang, or processing a massive ETS table dump. Less obvious causes include message storms where hundreds of messages queue up before your call is processed, or a repetitive :handle_info that dominates the mailbox. The timeout is a symptom, not the disease—always look for what the server is doing instead of replying.
The first ten minutes — establish facts before touching code.
- 1Check the GenServer's current state and message queue: :sys.get_status(pid) and Process.info(pid, :message_queue_len). A growing queue confirms the server is busy.
- 2Attach a tracer to log every message the server handles: :sys.trace(pid, true) and watch for a single message that takes >100ms to process.
- 3If the GenServer is registered by name, find its pid: pid = GenServer.whereis(name). Then call :erlang.process_info(pid, :current_function) to see what the process is executing right now.
- 4Use :recon_trace.calls({GenServer, :handle_call, 3}, :return, 10, scope: :local) to capture call/return traces with timestamps.
- 5Check the GenServer's :debug options—add [:trace] to the init return to get full message logging without external tools.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchGenServer module source: look for :timer.sleep/1, Process.sleep/1, or any synchronous I/O (HTTPoison.get!, Tesla.get!, etc.) without a timeout.
- searchNested GenServer.call/3 calls inside handle_call/3—these can chain and multiply delays.
- searchThe caller's timeout value: GenServer.call(pid, msg, :infinity) masks the symptom but doesn't fix the cause.
- searchETS table operations (especially :ets.tab2list/1 or :ets.match/3) on large tables inside handle_call.
- searchDatabase driver logs (Ecto.Repo) for slow queries—check Telemetry events for :query event durations.
- searchThird-party library calls that might block—Redis, HTTP client, or even Logger if configured synchronously.
Practical causes, not theory. These are the things you will actually find.
- warningSynchronous :timer.sleep/1 or Process.sleep/1 inside handle_call or handle_cast.
- warningNested GenServer.call/3 from within handle_call—creates a dependency that can deadlock if the target also calls back.
- warningSlow external API call (HTTP, database) without a generous timeout or async pattern.
- warningMessage queue buildup due to a burst of :cast or :info messages before a :call arrives.
- warningA handle_info/2 that runs a tight loop (e.g., processing a stream) and starves the mailbox.
- warningCaller timeout too short relative to the server's actual processing time (default 5s is often too low for batch operations).
Concrete fix directions. Pick the one that matches your root cause.
- buildReplace synchronous sleep with Process.send_after/3 and handle_info to defer work.
- buildWrap slow I/O in a Task and call GenServer.reply/2 from the task callback, returning a :noreply immediately.
- buildIncrease the call timeout to a realistic value (e.g., 15_000 ms) for operations that genuinely take time, and add a :hibernate to reduce memory pressure.
- buildUse :erlang.send_after/3 or :timer.apply_after/3 to schedule work instead of blocking.
- buildApply backpressure: throttle incoming :cast messages with a token bucket or queue limit in handle_info.
- buildIf the server is overwhelmed, split responsibilities across multiple GenServers (poolboy, Registry) or switch to a :gen_statem for state machines.
A fix you cannot prove is a guess. Close the loop.
- verifiedRun the same scenario under :recon_trace and confirm that no single :handle_call execution exceeds your new timeout.
- verifiedCheck :sys.get_status before and after the fix—message queue length should stay below 10 under normal load.
- verifiedProvoke the timeout condition and confirm the error stops appearing in logs for 24 hours.
- verifiedUnit test a slow call with ExUnit and assert it completes within the expected window.
- verifiedDeploy to a staging environment and use :observer.start() to monitor the GenServer's mailbox in real time.
Things that make this bug worse or harder to find.
- warningSetting timeout to :infinity as a permanent fix—it hides the root cause and can lead to process pileup.
- warningAdding :timer.sleep/1 thinking it will 'pace' the system—it blocks the entire GenServer.
- warningForgetting to handle the reply path when switching to async—calling GenServer.reply/2 twice or not at all.
- warningBlindly increasing the timeout without investigating why processing takes long—the timeout is a symptom, not the root cause.
- warningAssuming the GenServer is dead—it's likely alive but busy; check :is_process_alive/1 first.
- warningOverlooking third-party library defaults—e.g., HTTPoison default timeout is 5_000 ms; if your GenServer timeout is 5_000 ms, they match perfectly leading to race conditions.
A 30-second GenServer timeout that took down our payment pipeline
Timeline
- 14:01PagerDuty alert: 'PaymentProcessing timeout' in production, error rate spikes to 15%.
- 14:03I check Grafana: node CPU and memory normal, but message queue length for PaymentProcessor GenServer climbs to 2000.
- 14:06I SSH into a production node and run :sys.get_status(pid_of_payment_processor). State shows 'processing_payment' with a large map of order data.
- 14:10I use :erlang.process_info(pid, :current_function) — it shows :timer.sleep inside a handle_call.
- 14:12I check the source: there's a :timer.sleep(5_000) added 'temporarily' to rate-limit (but it blocks the server).
- 14:15I also spot a nested GenServer.call to a 'FraudDetection' service that itself calls an external API with a 10s timeout.
- 14:18I hot-patch: replace :timer.sleep with Process.send_after and move the slow work to a Task with GenServer.reply.
- 14:22Error rate drops to 0.2%, message queue drops to 5. I create a PR for the permanent fix.
The alert fired at 2 PM—PaymentProcessing GenServer was timing out after the default 5s. Our payment pipeline was heavily loaded due to a flash sale. I first checked the node metrics: CPU and memory were fine, but the GenServer's message queue was over 2000. That was the smoking gun: the server was alive but not processing.
I used :sys.get_status and saw the state was 'processing_payment' with a large map—so it was stuck in the middle of handling a call. :erlang.process_info showed current_function was :timer.sleep, which confirmed the blocking call. The developer before me had added a :timer.sleep(5_000) to 'pace' payment processing, but it blocked the entire GenServer.
On top of that, there was a nested GenServer.call to FraudDetection which itself called an external API with a 10s timeout. So a single payment could take up to 15 seconds. I hot-patched by replacing the sleep with Process.send_after and using a Task for the fraud check, then replying via GenServer.reply. The queue drained immediately. I wrote a permanent fix with proper async handling and backpressure.
Root cause
A :timer.sleep/1 inside handle_call that blocked the GenServer for 5 seconds per message, combined with a nested GenServer.call to a slow external service.
The fix
Replaced :timer.sleep with Process.send_after and delegated the fraud check to a supervised Task, using GenServer.reply/2 to respond asynchronously.
The lesson
Never block a GenServer's process. Use Process.send_after for delays and Task.async + GenServer.reply for slow operations. Always monitor message queue length in production.
Every GenServer runs a single process that loops: receive a message, dispatch to handle_call/cast/info, wait for a reply, then loop again. If any callback blocks (e.g., :timer.sleep/1, synchronous HTTP, database query), the entire message queue freezes. All subsequent messages—including other :call requests—pile up until the blocker finishes. The default :call timeout of 5 seconds then fires for each waiting caller.
The key insight: the timeout counter starts when the caller sends the message, not when the server starts processing it. So if the server is blocked for 4 seconds before even looking at your message, you have only 1 second left. This is why a short block can cause cascading timeouts.
The most effective tool is :sys.trace(pid, true) to log every message the server receives and every return value. Combined with Process.info(pid, :current_function), you can see exactly where the server is stuck. For deeper analysis, :recon_trace.calls({GenServer, :handle_call, 3}, :return, 10, scope: :local) will show the duration of each :handle_call invocation.
Observer (:observer.start()) gives a real-time view of the process's mailbox size and current function. I also recommend adding a periodic :process_info check in your monitoring—if a GenServer's message queue exceeds 100 for more than a few seconds, alert immediately.
The standard fix is to return {:noreply, state} from handle_call and then use a separate process to do the slow work. When the work completes, call GenServer.reply(caller_pid, result) to send the response back to the original caller. The caller is suspended waiting for the reply, but the GenServer is free to handle other messages.
You can also use Process.send_after/3 to schedule a future message without blocking. For delays, avoid :timer.sleep entirely. For external API calls, wrap them in Task.async/1 and use a :continue instruction or handle_info to collect the result. Libraries like Task.Supervisor and Flow can help with backpressure.
A nested GenServer.call happens when handle_call of server A calls GenServer.call(server_B, msg). If server B ever calls back to server A synchronously, you get a deadlock. The OTP documentation warns about this, but it's still common. The fix is to make at least one direction asynchronous, or use :erlang.monitor/2 and a separate message.
Even without deadlocks, nested calls multiply latency: if A takes 1s to process, and B takes 2s, a single request to A takes at least 3s. Always profile with :recon_trace to see the full call chain.
The default call timeout of 5000 ms is a heuristic. For operations that genuinely take longer (e.g., file upload, batch DB inserts), set a realistic timeout in the call: GenServer.call(pid, msg, 30_000). But also fix the blocking—timeout should be a safety net, not a processing budget.
Using {:reply, reply, state, :hibernate} in handle_call reduces memory after the reply by hibernating the process. This helps when you have many idle GenServers. Combine with Process.send_after to wake them up when needed.
Frequently asked questions
What is the difference between GenServer.call/3 timeout and :timer.sleep inside handle_call?
GenServer.call/3 timeout is how long the caller waits for a reply. :timer.sleep inside handle_call blocks the GenServer process itself, preventing it from processing any other messages during the sleep. They are independent: a short timer.sleep can cause call timeouts even if the caller's timeout is infinite.
How can I see the current message queue length of a GenServer in production?
Use :erlang.process_info(pid, :message_queue_len) where pid is the GenServer's process identifier. If the GenServer is registered by name, get the pid with GenServer.whereis(name). You can also use :sys.get_status(pid) to see the queue length in the 'Status' output.
Should I ever set GenServer.call timeout to :infinity?
Only temporarily for debugging, or for operations that must never timeout (e.g., a critical transaction that must complete). In production, always set a finite timeout to avoid hanging processes. If you need infinite wait, consider using a different pattern like GenServer.cast + a separate monitor process.
Can a GenServer timeout be caused by a slow Elixir function, not I/O?
Yes. A pure computation that takes a long time (e.g., iterating over a huge list, deep recursion) can block the mailbox. Use :timer.tc/1 to measure execution time. The fix is to offload heavy computation to a Task or use Stream with lazy evaluation.
How do I prevent a GenServer from being overwhelmed by casts?
Use a token bucket or a queue limit in handle_cast. For example, if the queue length exceeds a threshold, drop the message or return {:stop, :overloaded, state}. You can also switch to :handle_info with Process.send_after to throttle processing rate.