What this usually means
A thundering herd reconnect storm happens when a group of WebSocket clients all lose their connection at the same time (due to a server restart, network partition, or load balancer health check failure) and then immediately attempt to reconnect with the same exponential backoff schedule. Without jitter, clients synchronize their retry attempts, creating overlapping connection floods that overwhelm servers, exhaust file descriptors, and trigger cascading failures. The root cause is almost always missing or insufficient jitter in the reconnection backoff algorithm, combined with aggressive reconnect limits.
The first ten minutes — establish facts before touching code.
- 1Run `ss -s | grep -i estab` on a server node; if established connections drop sharply and then spike above normal, you're in a storm
- 2Check WebSocket server logs for repeated `Upgrade handshake` logs from the same client IP in under 1 second
- 3Look at client-side reconnect timestamps: if they're clustered within 100ms of each other, jitter is missing
- 4Inspect load balancer metrics (e.g., `aws elb describe-load-balancers` or `kubectl top pods`) for connection rate >5x baseline
- 5Check `ulimit -n` usage on servers: if file descriptors hit the soft limit during the storm, that's your bottleneck
The specific files, logs, configs, and dashboards that usually own this bug.
- search`/var/log/nginx/access.log` — look for repeated 101 responses to the same `$remote_addr` within seconds
- search`/var/log/syslog` or `journalctl -u websocket-server` — search for `connection refused` or `too many open files`
- searchClient-side reconnect configuration — check `reconnectTimeout`, `reconnectJitter` in source code or config files (e.g., `app.js`, `config.yaml`)
- searchLoad balancer health check settings — `interval`, `timeout`, `unhealthy_threshold` in Terraform or cloud console
- searchPrometheus metrics — `websocket_connections_total`, `connection_rate`, `goroutines` (Go) or `threads` (Java)
- searchKubernetes events — `kubectl describe pod <pod>` for OOMKill or CrashLoopBackOff events
Practical causes, not theory. These are the things you will actually find.
- warningExponential backoff without jitter — all clients retry at identical intervals (e.g., 1s, 2s, 4s, 8s...)
- warningAggressive reconnect limits — clients retry 5+ times before backing off, amplifying the storm
- warningSynchronized disconnection — all clients drop due to a single server restart or health check failure
- warningMissing connection rate limiting on the server side — no `max_connections_per_ip` or global rate limiter
- warningLoad balancer health check interval too short — causes rapid flapping of backends, triggering reconnects
- warningClient-side reconnect timeout too low — e.g., 500ms initial timeout with no jitter
Concrete fix directions. Pick the one that matches your root cause.
- buildAdd jitter to reconnect backoff: `Math.random() * baseTimeout` spread across [0, baseTimeout)
- buildImplement connection rate limiting on the server: e.g., `rate limit 10 req/s per IP` in nginx or a token bucket
- buildStagger client reconnections by using a random initial delay up to 5 seconds
- buildUse a backoff cap (e.g., max 30 seconds) to prevent infinite retry loops
- buildSet a global max connections per server and enforce via connection pool limits
- buildImplement a server-side 'connection drain' — when restarting, wait for existing connections to die naturally before new ones flood
A fix you cannot prove is a guess. Close the loop.
- verifiedDeploy the fix to a canary instance and simulate a simultaneous disconnect (e.g., restart the server) — monitor connection rate stays below 2x baseline
- verifiedCheck client reconnect timestamps: they should be spread across a 5-second window, not clustered
- verifiedRun `ab -n 1000 -c 100` (or equivalent) against the WebSocket endpoint to verify rate limiting works
- verifiedMonitor server CPU and file descriptor usage during a rolling restart — should stay under 80%
- verifiedUse tcpdump to capture WebSocket handshake packets: `tcpdump -i eth0 'tcp port 443' -w storm.pcap` and verify SYN rate is constant
Things that make this bug worse or harder to find.
- warningAdding jitter but not capping backoff — exponential growth still overwhelms servers after many retries
- warningRate limiting only on one server — if all servers have the same limit, the storm still hits each individually
- warningIgnoring client-side disconnect logic — if clients don't detect clean disconnects, they may not trigger reconnect at all (or trigger too late)
- warningSetting jitter too small (e.g., ±10ms) — synchrony still occurs; jitter should be at least 50% of the base timeout
- warningAssuming WebSocket libraries handle jitter by default — most do not; verify the source code
- warningOver-relying on server-side fixes without auditing client reconnect logic
The 3 AM Reconnect Storm That Took Down Our Chat Platform
Timeline
- 03:02PagerDuty alerts: all chat servers CPU >95%, latency >30s, 90% error rate
- 03:05Check Grafana: connection count dropped from 50k to 0 in 2 seconds, then spiked to 200k
- 03:08Check ALB metrics — TargetResponseTime >10s, healthy hosts going unhealthy
- 03:12SSH into a server: `ss -s` shows 15k connections, 10k in SYN_RECV, file descriptors at 99%
- 03:15Found a new deployment went out at 03:00 — rolling restart caused all pods to cycle
- 03:20Client code review: reconnection uses `time.Sleep(1*time.Second)` — no jitter, no cap
- 03:30Hotfix: added random jitter (±500ms) and cap at 30s, deployed to 10% canary
- 03:35Canary stable; rolled out to all clusters
- 03:50All metrics back to baseline; no further storms
At 3 AM, a routine deployment triggered a rolling restart of our Go WebSocket servers. The moment the first pod went down, all 50,000 connected clients lost their WebSocket connection simultaneously. The client library — a homegrown JavaScript wrapper — had a hardcoded `setTimeout(reconnect, 1000)` with zero jitter. Every single client fired a new connection request exactly 1 second later. Our ALB, configured with a 2-second health check interval, saw the storm as a healthy surge and routed traffic to the remaining pods. Those pods instantly hit 15,000 connections each (our ulimit was 16k). They started rejecting new connections with `connection refused`, which triggered more reconnect attempts from clients. Within 30 seconds, all 6 pods were at 100% CPU, file descriptors exhausted, and the ALB marked them unhealthy — which caused more disconnects. A classic thundering herd.
I jumped on the box and ran `ss -s` — established connections were dropping, but SYN_RECV was through the roof. The kernel was drowning in incoming TCP SYN packets. I checked `ulimit -n` and saw we were at 15,995 of 16,384. The server logs were a wall of `accept: too many open files`. I killed the process, but the ALB health check kept sending requests. We had to temporarily block the health check IP to get the server stable.
Once we had a breather, I pulled the client source code. There it was: `function reconnect() { setTimeout(connect, 1000); }`. No jitter, no backoff, no cap. I added `let jitter = Math.random() * 1000; let delay = Math.min(1000 + jitter, 30000);`. Deployed to a single canary pod, watched the connection rate stay flat, then rolled to all. Lesson: always add jitter. Always. And never trust client libraries to handle reconnect gracefully.
Root cause
Missing jitter in client reconnect logic — all 50k clients reconnected at the same 1-second interval, causing SYN flood and file descriptor exhaustion.
The fix
Added random jitter (±500ms) and a 30-second cap to the client reconnect timeout. Also added server-side rate limiting (max 100 connections/sec per IP) in nginx.
The lesson
Never deploy a WebSocket client without jitter in the reconnect logic. Always simulate a simultaneous disconnection in staging. Monitor file descriptors and connection rate as critical metrics.
A thundering herd occurs when many clients all react to the same event (e.g., server restart) by trying to reconnect at the same time. Without jitter, exponential backoff algorithms like `1s, 2s, 4s, 8s` are synchronized across clients because they all start their timer at the same instant. The result: every client retries simultaneously, creating a spike of new connections that can exceed server capacity by 10x or more.
The problem compounds when servers under load reject connections — clients interpret rejections as 'server down' and retry immediately, often with a shorter timeout. This feedback loop can collapse a cluster in seconds. The key metric to watch is the rate of incoming SYN packets vs. the server's SYN backlog (net.ipv4.tcp_max_syn_backlog). If SYN_RECV connections spike while established connections drop, you're in a storm.
The most effective jitter is a uniform random delay added to the base backoff: `delay = base + random(0, jitterWindow)`. The jitter window should be at least equal to the base timeout — for example, if base is 1s, jitter should be at least 1s. This spreads reconnects across a full window. A common mistake is using a tiny jitter (e.g., ±100ms) which doesn't break synchrony for large client counts.
For capped exponential backoff, use `delay = min(cap, base * 2^attempt) + random(0, jitter)`. The cap prevents unbounded growth. Some implementations use 'full jitter' where the delay is random between 0 and the full exponential value — this works but can cause very long waits for some clients. The AWS SDK, for instance, uses a combination of exponential backoff and jitter with a cap.
While client-side jitter is the first line of defense, server-side measures are essential as a safety net. nginx can limit connection rate per IP: `limit_conn_zone $binary_remote_addr zone=addr:10m; limit_conn addr 10;`. For WebSocket upgrades, use `limit_req zone=websocket burst=20 nodelay;`. In Kubernetes, consider a `NetworkPolicy` that limits ingress connections per pod.
Another technique is 'connection draining' — when a server receives a SIGTERM for a rolling restart, it should stop accepting new connections immediately and wait for existing ones to close naturally (up to a timeout). This prevents the 'all at once' disconnection trigger. In Go, you can use `http.Server`'s `Shutdown()` method with a graceful period.
The fastest detection is a sudden spike in TCP connection rate. Use `sar -n TCP 1` to watch for `active/s` (client connects) — a value >1000/s is suspicious for a typical server. On the load balancer, monitor `NewConnectionCount` (ALB) or `backend_connection_attempts` (nginx). A 10x increase from baseline is a storm.
Client-side, instrument the reconnect function to emit metrics: `reconnect_attempt_total`, `reconnect_delay_seconds`. If you see a tight cluster of delays (e.g., all within 50ms of each other), jitter is missing. Correlation with deployment events is key — check your CI/CD pipeline logs for the exact time of a rolling restart.
Frequently asked questions
Why does exponential backoff alone not prevent thundering herd?
Exponential backoff spreads retries over time, but if all clients start at the same time and use the same algorithm, they will retry at the same intervals (1s, 2s, 4s...). The synchrony persists. Jitter adds randomness so that each client's timeline diverges.
What is the ideal jitter value?
A good rule: jitter should be at least 50% of the base timeout. For a 1-second base, use a jitter window of 0.5–1 second. This ensures enough spread to avoid overlapping retries. Some systems use 'full jitter' where the entire delay is random between 0 and the exponential value.
How do I simulate a reconnect storm in staging?
Use a load testing tool like `k6` or `wrk` to open many WebSocket connections, then restart the server (kill -9 the process). Monitor connection rate and SYN backlog. To test jitter, check that reconnect timestamps are distributed uniformly over the jitter window.
Can server-side rate limiting alone fix the storm?
No. Rate limiting can protect the server from overload, but it will still reject many clients, causing them to retry and potentially enter a tighter retry loop. Client-side jitter is necessary to prevent the storm from happening in the first place.
What is the difference between a reconnect storm and a DDoS attack?
A reconnect storm originates from legitimate clients reacting to a server event. A DDoS is malicious. The symptoms (connection spike, high CPU) look similar, but a storm's source IPs are your real user base, while a DDoS often comes from many random IPs. Check client IP distribution: storm IPs will match your user IPs.