Retries are the duct tape of distributed systems. A request fails? Retry it. The database is slow? Retry the query. A downstream service returns a 503? Retry with a short sleep. On its own, each retry seems harmless — maybe even helpful. But when a system starts to fail under load, retries are often the thing that turns a minor blip into a full-blown outage.
I've seen this pattern play out more times than I'd like to admit. It's called a retry storm, and it's one of the most common — and most preventable — causes of cascading failures in distributed systems.
How Retries Create a Feedback Loop
Imagine a service A that depends on service B. B starts to slow down — maybe because of a noisy neighbor, a bad deploy, or a traffic spike. A's requests to B start timing out. A's retry logic kicks in: it waits 100ms and tries again. Meanwhile, all other instances of A are doing the same thing. The number of requests hitting B doubles, then triples. B gets slower, more requests time out, and the retries pile up.
Now extend this to a chain. Service A calls B, B calls C, and C calls D. If D hiccups, C retries, holding connections open. B's pool of connections to C fills up, so B's requests to C start timing out. B retries, holding connections to A. A's pool fills up, and now A can't serve its own clients. The failure cascades up the stack, and everyone is just retrying into a black hole.
A retry storm can multiply load by an order of magnitude in seconds. A service that was at 60% CPU can go to 600% if every client retries once. No amount of autoscaling can keep up with that.
A Real Incident: The 10x Retry Multiplier
The Database Connection Pool Collapse
- 14:02Primary database node fails over to replica due to hardware issue.
- 14:03Replica takes 3 seconds to accept connections. All 50 app servers' connection pools time out.
- 14:04Each app server retries its queued queries with 50ms backoff. 50 servers × 20 threads each = 1000 retries/s hitting the replica.
- 14:05Replica CPU spikes to 95%. Query latency jumps from 5ms to 500ms.
- 14:06App servers see slower responses, open more connections to retry. Connection pool on replica exhausts at 200 connections.
- 14:07All new requests fail with 'connection refused'. Full outage begins.
- 14:12Engineers disable retry logic and restart app servers. Service recovers in 5 minutes.
Lesson
The retry logic had no jitter and no circuit breaker. All servers retried in nearly the same pattern, creating a coordinated storm. Adding exponential backoff with jitter and a circuit breaker would have limited the blast radius to a few seconds of degraded performance instead of a 10-minute outage.
The Anatomy of a Retry Storm
- arrow_rightMultiple clients (or threads) detect failure at roughly the same time.
- arrow_rightThey all retry after a fixed or short random interval.
- arrow_rightThe retries arrive as a synchronized wave, spiking server load.
- arrow_rightServer becomes slower, causing more timeouts and more retries.
- arrow_rightConnection pools, thread pools, and queues fill up, making the problem worse.
- arrow_rightThe cascade propagates to upstream services as they wait on downstream retries.
Defense #1: Exponential Backoff with Jitter
The single most effective countermeasure is to make retries happen at unpredictable intervals. Exponential backoff doubles the wait time after each retry (e.g., 100ms, 200ms, 400ms). But without jitter, all clients with the same retry count will retry at the same time. The fix is to add random jitter.
import random
import time
def retry_with_backoff(attempt, base_delay=0.1, max_delay=10.0):
delay = min(base_delay * (2 ** attempt), max_delay)
# Add full jitter: random between 0 and delay
sleep_time = random.uniform(0, delay)
time.sleep(sleep_time)
return sleep_timeI prefer 'full jitter' because it spreads retry times evenly. Some systems use 'equal jitter' (delay ± half), but I've seen that still create clustering. Full jitter is simpler and works well in practice.
Defense #2: Circuit Breakers
Backoff alone isn't enough when the downstream is truly down. A circuit breaker stops retries entirely once a failure threshold is crossed. In the incident above, a circuit breaker would have tripped after, say, 5 consecutive failures, and all retries would stop for 30 seconds. That gives the database time to recover without drowning it in requests.
import pybreaker
breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30)
def call_database(query):
with breaker:
# actual database call
return db.execute(query)Defense #3: Retry Budgets and Client-Side Rate Limiting
Sometimes a circuit breaker is too coarse — you might want to allow some retries but not a flood. A retry budget limits the total number of retries a client can issue per minute. For example, a client might be allowed 100 retries per minute. If it exhausts that budget, retries are rejected until the next window.
This is especially useful in multi-tenant systems where one noisy client should not affect others. It also prevents retry amplification when a single client has many threads.
Load amplification observed in a real retry storm when 10 clients retried 3 times each with no backoff.
Idempotency and Safety
Retries are only safe if the operation is idempotent — meaning repeating it has the same effect as doing it once. If a retry causes a duplicate payment, a double write, or a duplicated email, you've traded one failure for a worse one. Always design your APIs to be idempotent (e.g., using idempotency keys) before adding retry logic.
Idempotency keys: a client sends a unique key with each request. The server deduplicates based on that key. This makes retries safe even if the first request actually succeeded but the response was lost.
Monitoring for Retry Storms
You can't fix what you don't see. Add metrics for retry counts, retry rates, and backoff durations. Track the ratio of retries to initial requests. If that ratio spikes above 1.0, something is wrong. Also monitor circuit breaker state transitions: if a breaker opens, you should get an alert.
At the infrastructure level, watch for connection pool exhaustion and thread pool saturation. Those are often the first signs of a retry storm.
# Example prometheus query for retry ratio
rate(http_requests_total{status="retry"}[1m]) / rate(http_requests_total{status="initial"}[1m]) > 0.5Putting It All Together
- 1Implement exponential backoff with full jitter on every retry.
- 2Add a circuit breaker that stops retries after a small number of failures.
- 3Set per-client retry budgets to limit blast radius.
- 4Make all retried operations idempotent.
- 5Monitor retry rates and circuit breaker states.
- 6Test retry behavior under failure conditions (chaos engineering).
The next time you add a retry to your code, ask yourself: what happens if everyone retries at the same time? If the answer is 'bad things', then you need backoff, jitter, and a circuit breaker. Retries are not free — they are a form of load. Treat them like one.
Frequently asked questions
What exactly is a retry storm?
A retry storm is a feedback loop where many clients retry failed requests at the same time, creating a spike in load that overwhelms the server, causing more failures and more retries. It's a common cause of cascading failures.
How does exponential backoff help prevent retry storms?
Exponential backoff spaces out retries by doubling the wait time after each failure. Adding random jitter prevents clients from retrying in lockstep, which would otherwise create synchronized waves of retries.
What's the difference between a circuit breaker and a retry budget?
A circuit breaker stops all retries to a failing service for a period once a failure threshold is crossed. A retry budget limits the total number of retries a client can perform over a time window. Both prevent uncontrolled retries.
Can retry storms occur with database queries?
Yes. Connection pools and ORM layers often retry queries on timeout. If a database becomes slow, clients retry, opening more connections, exhausting the connection pool, and causing a cascading failure.