I've lost count of the times I've seen a production incident traced back to a single root cause: a misconfigured timeout. Not a bug in the code, not a hardware failure — just a magic number someone copy-pasted from Stack Overflow five years ago. Timeouts are the silent killers of distributed systems. They don't crash your service; they just make everything slow and unreliable until someone notices the p99 latency graph looks like a roller coaster.
The problem is that most engineers treat timeouts as constants. You define `TIMEOUT = 30` and move on. But timeouts are not constants — they are contracts between your service and its dependencies. And like any contract, if you don't read the fine print, you'll get sued. This article covers the concrete mistakes I've seen (and made) and how to fix them.
Mistake #1: Using a Single Timeout for Everything
Most HTTP clients and database drivers let you set a single timeout. Resist the urge. A connection timeout and a read timeout protect against different failure scenarios. If you use one value, you might wait 30 seconds for a connection that will never succeed, or you might abort a request that simply has a slow stream of data.
Here's a concrete example: We had a service that called an external API with a 10-second timeout. The API occasionally took 12 seconds to respond. Instead of timing out after 10 seconds, the connection would hang because the TCP keepalive wasn't configured. The single timeout was applied to the entire operation, but the connection phase succeeded in 200ms, then the read phase waited 9.8 seconds before the server sent a single byte. The timeout never fired because the clock reset on each data chunk. The fix: separate connection timeout (5s) and read timeout (5s).
A single timeout is a lie. You need at least three: connection, read, and write.
import requests
from urllib3.util import Timeout
# Bad: single timeout
response = requests.get('https://slow-api.example.com/data', timeout=30)
# Good: separate timeouts
timeout = Timeout(connect=5.0, read=5.0)
response = requests.get('https://slow-api.example.com/data', timeout=timeout)Mistake #2: Ignoring Network Latency Variability
Setting a timeout based on your local development environment is a rookie mistake. In production, network round-trip times (RTT) can vary by orders of magnitude due to cross-region deployments, load balancers, or CDN routing. A timeout that works in us-east-1 will fail in eu-west-1 if you don't account for the 70ms baseline difference.
I once worked on a system where the database was in a different AWS region from the application servers. The timeout was set to 100ms — fine for local, but the cross-region RTT was 80ms. The database queries themselves took 50ms, so total time was 130ms. Every third request timed out. The fix: set timeouts based on measured p99 latency from the client's perspective, not from a dashboard in the same datacenter.
Measure latency from the client, not the server. Use tools like `mtr` or `tcpping` to get real RTTs. Then add 50% headroom for spikes.
How to Measure the Right Value
For each dependency, collect a histogram of response times over a week. Use the p99 value (99th percentile) as your baseline. Then add the network RTT and a buffer for transient spikes. The formula I use: `timeout = (p99_latency + avg_rtt) * 1.3`. If the p99 is 200ms and RTT is 50ms, that's 325ms. Round up to 350ms for simplicity.
But don't stop there. Monitor timeout rate over time. If you see timeouts increasing, it's a signal that either the dependency is degrading or your network path has changed. Set up alerts when timeout rate exceeds 0.1% of requests.
The 3-Second Timeout That Took Down Search
- 14:02Search latency spikes to 10s; users report errors.
- 14:05On-call engineer sees connection pool exhaustion on the search service.
- 14:12Root cause: The recommendation service had a 3-second timeout for a search call. Under normal load, search responded in 200ms. But a deployment had a memory leak causing GC pauses up to 2.5s.
- 14:20Because the timeout was 3s, 90% of requests still succeeded, but each request held a connection for 3s. The pool of 50 connections was exhausted, causing all other requests to queue.
- 14:30Fix: reduce timeout to 500ms and add circuit breaker. The recommendation service now fails fast instead of dragging down search.
Lesson
A timeout that is too long can exhaust connection pools and cause cascading failures. Fail fast with aggressive timeouts, then use retries with backoff for transient errors.
Mistake #3: Not Coordinating Timeouts with Retries
Retries are the natural complement to timeouts, but they multiply the total wait time. If you have a 5-second timeout and 3 retries, the caller might wait 20 seconds before giving up. That's unacceptable for user-facing services. You need a total retry budget that fits within your overall service-level objective (SLO).
Here's the pattern I use: define a maximum total time the caller is willing to wait (e.g., 2 seconds for a user-facing API). Distribute that budget across attempts with exponential backoff. For example, with 3 attempts: attempt 1: 500ms timeout, attempt 2: 700ms (after 200ms backoff), attempt 3: 1s (after 500ms backoff). Total worst-case: 500 + 200 + 700 + 500 + 1000 = 2.9s. Still above 2s, so I'd reduce the timeouts or limit to 2 retries.
import time
import random
def call_with_retry(url, max_total_time=2.0, max_attempts=3):
attempt = 1
total_elapsed = 0.0
while attempt <= max_attempts:
start = time.monotonic()
try:
# Per-attempt timeout = remaining budget * 0.6
timeout = (max_total_time - total_elapsed) * 0.6
response = requests.get(url, timeout=timeout)
return response
except requests.Timeout:
elapsed = time.monotonic() - start
total_elapsed += elapsed
if total_elapsed >= max_total_time:
raise
# Exponential backoff with jitter
backoff = min(0.1 * (2 ** attempt), 1.0) * random.uniform(0.5, 1.5)
time.sleep(backoff)
attempt += 1
raise Exception("Max retries exceeded")Mistake #4: Forgetting About Queueing Time
Timeouts apply to the time between sending a request and receiving a response. But if your service uses a thread pool or event loop, the request may spend time waiting in a queue before it's even processed. That queuing time counts toward your timeout. I've seen systems where the thread pool is too small, so requests pile up, and the timeout expires before the request is even handled. The client sees a timeout, but the server never saw the request — it's a false negative.
The fix: either increase the thread pool size to avoid queuing, or set a separate queue timeout that is shorter than the network timeout. In async systems, use a bounded queue with a timeout on the `enqueue` operation. If the queue is full, fail fast rather than letting the request sit.
of timeout-related incidents in our postmortem database involved queueing delays, not actual network or server slowness.
Mistake #5: Not Instrumenting Timeout Failures
Most code catches timeout exceptions and logs a generic "request timed out". That's useless. You need to know: which endpoint? how long did it wait? was it the connection or read phase? what was the server's latency at the time? Without this data, you can't distinguish between a transient network blip and a systemic problem.
I advocate for structured logging with fields: `timeout_type`, `configured_timeout`, `actual_wait_time`, `target_service`, `attempt_number`. Then you can build dashboards showing timeout rates per endpoint and percentile of actual wait times before timeout. This data is gold when tuning timeouts.
{
"level": "warn",
"message": "Request timed out",
"timeout_type": "read",
"configured_timeout_ms": 5000,
"actual_wait_ms": 5012,
"target_service": "payment-api",
"endpoint": "/charge",
"attempt": 1,
"trace_id": "abc123"
}Testing Timeouts Under Chaos
Unit tests can't catch timeout misconfigurations. You need integration tests that inject latency and packet loss. My go-to tool is Toxiproxy — it's a TCP proxy that can simulate network failures. I add Toxiproxy between my service and its dependencies in staging, and I run a suite of tests with various latency profiles.
Here's a simple test: configure Toxiproxy to add 2 seconds of latency to the database connection. Then verify that your service's timeout fires within 500ms (if that's your configured read timeout). If it waits 2 seconds, you know the timeout isn't being applied correctly.
{
"name": "db_latency",
"type": "latency",
"toxicity": 1.0,
"attributes": {
"latency": 2000,
"jitter": 100
}
}Putting It All Together
Here's my checklist for configuring timeouts in any distributed system:
1. Separate connection, read, and write timeouts. Never use a single value.
2. Measure p99 latency from the client, add network RTT, and multiply by 1.3.
3. Set a total retry budget that fits within your SLO, and distribute timeouts across attempts with exponential backoff and jitter.
4. Account for queueing time inside your service — use bounded queues with timeouts.
5. Log timeout failures with structured fields. Build dashboards to monitor trends.
6. Test with chaos engineering tools like Toxiproxy to validate behavior under real-world conditions.
Timeouts are not a one-and-done configuration. They require ongoing measurement and tuning. But if you treat them as a critical part of your system's reliability, you'll avoid the class of incidents that keep engineers awake at 3 AM.
Frequently asked questions
What is the difference between a connection timeout and a read timeout?
A connection timeout is the maximum time allowed to establish a TCP connection (including TLS handshake). A read timeout is the maximum time to wait for data after the connection is established. They protect against different failure modes: connection timeouts guard against unreachable hosts; read timeouts guard against slow responses or hung servers.
How do I calculate a reasonable timeout value for an HTTP API call?
Start with the target service's p99 latency under normal load. Add the expected network round-trip time (RTT) and a safety margin of 30–50%. Then multiply by your maximum retry count to get the total wait time before giving up. For example, if p99 is 200ms and RTT is 50ms, set a timeout of ~375ms per attempt (200+50)*1.5, and with 3 retries, total budget ~1.1s.
Why shouldn't I set a very high timeout to avoid failures?
High timeouts mask real performance problems, allowing slow or broken dependencies to consume resources (threads, connections, memory) until they exhaust pool limits. This can cause cascading failures where one slow service brings down many callers. Timeouts should be aggressive enough to fail fast and trigger circuit breakers early.
How do timeouts interact with retries?
Retries multiply the total time a request can take. If you retry 3 times with a 5-second timeout each, the caller may wait 15+ seconds before failing. This can exhaust client-side timeouts and queue up work. Always define a total retry budget (e.g., 2 seconds) and distribute it across attempts, using exponential backoff with jitter.