What this usually means
Each AMQP connection in RabbitMQ is backed by an Erlang process. Opening a connection involves TCP handshake, SSL negotiation (if used), authentication, and channel setup — all CPU-intensive. When clients tear down and reconnect at high frequency (connection churn), the Erlang VM spends more time spawning and garbage-collecting processes than handling actual message traffic. The root cause is almost always on the client side: a misconfigured connection pool that creates too many short-lived connections, a heartbeat timeout that kicks in prematurely (e.g., due to network latency or long message processing), or a framework/library that recreates connections on every operation instead of reusing them. In rare cases, a load balancer with aggressive idle timeout can force clients to reconnect constantly.
The first ten minutes — establish facts before touching code.
- 1Run `rabbitmqctl list_connections name state connected_at` and sort by connected_at — look for connections that last less than 60 seconds.
- 2Check `rabbitmq-diagnostics environment` for `heartbeat` value (default 60). If clients set a lower heartbeat, they may disconnect prematurely.
- 3Examine client-side metrics: connection open/close rate per second. In Prometheus, look at `rabbitmq_connections_opened_total` and `closed_total` — a high delta indicates churn.
- 4Inspect the client library logs for repeated 'connection lost' followed by 'reconnecting'. Note the time between reconnect attempts.
- 5If using a load balancer, check its idle timeout settings. Any LB timeout shorter than the heartbeat interval will cause silent connection drops.
- 6Use `strace -e trace=network -p <rabbitmq-beam-pid>` to see accept/close syscall rate — hundreds per minute confirms churn.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchRabbitMQ logs: `/var/log/rabbitmq/rabbit@hostname.log` — search for 'missed heartbeats' or 'connection closed'
- searchClient application logs: look for 'AMQPConnectionError', 'channel closed', 'reconnecting'
- search`rabbitmqctl list_connections` output: capture `name`, `state`, `channels`, `connected_at`, `timeout`
- searchLoad balancer config (HAProxy, Nginx, AWS NLB): check `timeout client` and `timeout server` settings
- searchClient library configuration: examine `heartbeat`, `connection_timeout`, `requested_heartbeat`, `automatic_recovery` settings
- searchErlang process count: `rabbitmq-diagnostics erlang_processes` — compare to baseline (usually < 2000 per vhost)
- search`/proc/<rabbitmq-pid>/status` — check `voluntary_ctxt_switches` and `nonvoluntary_ctxt_switches` for context switch rate
Practical causes, not theory. These are the things you will actually find.
- warningClient heartbeat interval set too low (e.g., 1 second) causing premature timeouts under load
- warningLoad balancer idle timeout shorter than RabbitMQ heartbeat interval, forcing TCP resets
- warningClient library connection pooling misconfigured — creating a new connection per message instead of reusing
- warningNetwork latency spikes causing heartbeats to arrive late — especially in cross-region deployments
- warningFirewall or proxy dropping idle TCP connections after a fixed idle time (e.g., AWS NLB default 350s)
- warningClient application threading bug: threads create connections and never close them properly, leading to resource exhaustion and forced restarts
Concrete fix directions. Pick the one that matches your root cause.
- buildSet client heartbeat to at least 10 seconds, preferably 30-60s, and ensure it's lower than any network middlebox timeout
- buildIf behind a load balancer, set LB idle timeout to at least 2x the heartbeat interval (e.g., LB timeout 120s, heartbeat 60s)
- buildUse connection pooling: reuse connections across threads; most client libraries offer PooledConnectionFactory or similar
- buildEnable automatic recovery in the client (e.g., `factory.setAutomaticRecoveryEnabled(true)` in Java) to avoid manual reconnects
- buildIncrease TCP keepalive: `sudo sysctl -w net.ipv4.tcp_keepalive_time=300` to detect dead connections without heartbeats
- buildIf using a proxy like HAProxy, set `timeout client 180s` and `timeout server 180s` and enable `option tcpka`
A fix you cannot prove is a guess. Close the loop.
- verifiedMonitor CPU after fix: `top -p <rabbitmq-pid>` — should drop to <30% under normal load
- verifiedCheck connection lifetime: `rabbitmqctl list_connections name connected_at` — connections should live for hours, not seconds
- verifiedPlot `rabbitmq_connections_opened_total` and `closed_total` over 10 minutes — rate should be near zero after initial ramp-up
- verifiedRun a soak test: send steady message traffic for 30 minutes; connection count should stay flat
- verifiedCheck Erlang process count: `rabbitmq-diagnostics erlang_processes` — should stabilize, not grow unbounded
Things that make this bug worse or harder to find.
- warningDon't increase heartbeat interval without also checking load balancer timeout — they must be aligned
- warningDon't disable heartbeats entirely (`heartbeat=0`) — this can hide dead connections and cause resource leaks
- warningDon't blindly increase Erlang process limit (`+P` flag) without fixing the root cause — you'll just delay OOM
- warningDon't assume it's a server problem — 90% of connection churn issues are client-side misconfiguration
- warningDon't restart RabbitMQ to clear connections — this kills all connections and can cause a thundering herd on restart
The 3 AM Heartbeat Storm
Timeline
- 02:15PagerDuty alert: RabbitMQ CPU > 95% on prod cluster
- 02:18SSH to node; `top` shows beam.smp at 98% CPU
- 02:20`rabbitmqctl list_connections` shows 500+ connections, many with age < 10s
- 02:22Check HAProxy stats: `timeout client` set to 30s, connections reset after idle
- 02:25Java client logs show repeated 'heartbeat missed' every 30s
- 02:30Client heartbeat set to 60s; LB timeout 30s causes TCP RST before heartbeat arrives
- 02:35Fix: increase HAProxy timeout to 120s and set client heartbeat to 30s
- 02:40CPU drops to 20%; connections stabilize at 200
I got the alert at 2:15 AM. The RabbitMQ node was pegged at 98% CPU, and the on-call runbook just said 'check connections.' I ran `rabbitmqctl list_connections` and saw a flood of connections lasting only a few seconds each — classic churn. The Erlang process count was over 10,000, when normally it's around 1,500.
I checked HAProxy config first because we'd changed it last week. There it was: `timeout client 30s`. Our Java clients had heartbeat set to 60 seconds, but the LB was killing idle connections after 30s. The client would then reconnect, but the heartbeat timer reset, and after another 30s idle it got killed again. This cycle created ~20 new connections per second per instance.
I fixed the HAProxy timeout to 120s and aligned the client heartbeat to 30s. CPU dropped immediately. The lesson: when you change network infrastructure (LB, firewall), you must verify it plays nice with your messaging heartbeat settings. Now we monitor connection churn rate (opens - closes per second) as a standard alert.
Root cause
HAProxy idle timeout (30s) shorter than client heartbeat interval (60s), causing cyclic connection drops and reconnects.
The fix
Increased HAProxy `timeout client` to 120s and reduced client heartbeat to 30s (ensuring heartbeat can be delivered before LB timeout).
The lesson
Always align load balancer timeouts with application heartbeats — the LB timeout should be at least 2x the heartbeat interval. Monitor connection churn rate as a proactive metric.
Each AMQP connection in RabbitMQ is an Erlang process. Opening a connection involves TCP handshake, SSL handshake (if used), authentication via the internal database or LDAP, and channel negotiation. The Erlang VM must spawn a new process, allocate memory for its mailbox and heap, and perform garbage collection when the connection closes. With churn rates above 10 connections per second per node, the VM spends more time in process management than in message routing.
In extreme cases, the atom table can grow if connection names (often random) are generated and never garbage-collected until a full sweep. This leads to memory pressure and eventually out-of-atom errors. The CPU spike is not from message processing but from context switching between thousands of short-lived Erlang processes.
Hit `GET /api/connections` and parse the JSON. Look for connections with short `connected_at` timestamps. The field `timeout` shows the negotiated heartbeat (0 if disabled). Use `jq` to sort: `curl -s http://localhost:15672/api/connections | jq '.[] | {name, connected_at, timeout, channels}' | head -20`.
Also check `GET /api/nodes/{node}` for `proc_used` and `run_queue`. A high `run_queue` (e.g., > 100) indicates the Erlang scheduler is overloaded, consistent with churn-driven CPU.
In Spring Boot, the default `CachingConnectionFactory` creates a single connection by default. However, if you set `cacheMode=CONNECTION` with a small `connectionCacheSize`, it may create multiple short-lived connections. Ensure `cacheMode=CHANNEL` and `connectionCacheSize=1` unless you have a specific need for multiple connections.
Heartbeat is configured via `spring.rabbitmq.requested-heartbeat=30`. Also set `spring.rabbitmq.connection-timeout=5000` (ms) to avoid hanging on slow networks. Enable automatic recovery: `spring.rabbitmq.automatic-recovery.enabled=true` with `spring.rabbitmq.recovery-interval=10000`.
For HAProxy, set `timeout client 180s`, `timeout server 180s`, and `timeout connect 5s`. Add `option tcpka` to enable TCP keepalive. This ensures the LB respects the TCP connection state and doesn't kill idle connections prematurely.
For AWS NLB, the default idle timeout is 350 seconds. If your heartbeat is higher than that (e.g., 600s), you'll see churn. You cannot change NLB idle timeout — you must reduce the client heartbeat to <= 350s. Alternatively, use an ALB with HTTP health checks, but that's less efficient for AMQP.
Track `rabbitmq_connections_opened_total` and `closed_total` as rate counters in Prometheus. A sustained rate above 0.1 per second per node warrants investigation. Set alerts for `rate(rabbitmq_connections_opened_total[5m]) > 1`.
Also monitor `rabbitmq_erlang_processes` — a steady increase over hours indicates a leak. Correlate with connection churn events in your incident management tool.
Frequently asked questions
What is a normal connection open/close rate for RabbitMQ?
In a stable system, connections are opened once and stay open indefinitely. The open/close rate should be near zero after initial startup. A rate above 1 connection per second is abnormal and likely indicates churn.
Can a single client cause connection churn that affects other clients?
Yes. If one client creates thousands of short-lived connections, it can exhaust Erlang process slots or CPU, causing heartbeats for all clients to be delayed and leading to cascading disconnections. RabbitMQ has per-connection limits, but churn from one source can still degrade overall performance.
Should I set heartbeat to 0 to avoid timeouts?
No. Disabling heartbeats entirely means dead connections will not be detected, leading to resource leaks (file descriptors, memory). It's better to set a reasonable heartbeat (10-60s) and ensure network infrastructure doesn't kill idle connections.
How do I test if my LB timeout is causing churn?
Temporarily bypass the LB and connect clients directly to RabbitMQ. If churn stops, the LB is the culprit. Alternatively, set heartbeat to a very low value (e.g., 5s) and see if churn decreases (because heartbeats arrive before LB timeout).
Does connection churn affect message delivery semantics?
Yes. If a connection drops while a message is being published, the publisher may get a connection error and need to retry. Consumers may miss messages if they haven't acknowledged them. Using publisher confirms and consumer acknowledgements can mitigate, but churn increases the risk of duplicate or lost messages.