What this usually means
This error occurs when one side of a TCP connection sends a RST packet, abruptly terminating the connection. In the context of HTTP keep-alive, it typically means the server (or an intermediate proxy) has closed the connection while the client still believes it's open and attempts to reuse it. The root cause is often a timeout mismatch: the server's keep-alive timeout expires and it closes the socket, but the client hasn't yet received the FIN because of buffering or a proxy that delayed forwarding. Alternatively, the server process crashed or was killed, or a firewall or load balancer sent a RST due to idle timeout. Non-obvious causes include server-side connection pool exhaustion (leading to proactive RST), HTTP/2 downgrade proxies that misbehave, and kernel TCP keepalive settings overriding application-level timeouts.
The first ten minutes — establish facts before touching code.
- 1Run `ss -tanpo` on the server to see socket states; look for CLOSE_WAIT or TIME_WAIT sockets accumulating.
- 2Capture traffic with `tcpdump -i eth0 -s0 port 80 -w keepalive.pcap` and analyze with Wireshark, filtering for TCP RST.
- 3Check server error logs for 'connection reset by peer' and correlate timestamps with client-side retries.
- 4Verify keep-alive timeout settings: nginx `keepalive_timeout`, Apache `KeepAliveTimeout`, HAProxy `timeout http-keep-alive`.
- 5Test with a simple keep-alive client: `curl -v --keepalive-time 1 http://your-server/` and observe RST after idle.
- 6Check kernel TCP keepalive parameters: `sysctl net.ipv4.tcp_keepalive_time` and compare with application timeouts.
The specific files, logs, configs, and dashboards that usually own this bug.
- search/var/log/nginx/error.log or /var/log/httpd/error_log for 'connection reset by peer'
- search`/proc/net/tcp` and `/proc/net/tcp6` to inspect TCP connection states
- search`tcpdump -i any port 80 or port 443 -w capture.pcap` for packet-level analysis
- searchApplication server logs (e.g., Java stdout, Python gunicorn logs) for stack traces
- searchLoad balancer access logs (e.g., HAProxy stats page, AWS ALB logs)
- search`ss -tanpo | grep CLOSE_WAIT` to identify sockets stuck in half-closed state
- search`strace -e trace=network -p <pid>` to see syscalls returning ECONNRESET
Practical causes, not theory. These are the things you will actually find.
- warningServer-side keepalive_timeout shorter than client idle interval, causing server to close idle socket while client reuses it
- warningIntermediate proxy or load balancer with idle timeout (e.g., AWS ALB 60s) that sends RST before the server
- warningServer application crash or restart that resets all open connections (e.g., OOM killer, deployment rolling restart)
- warningConnection pool exhaustion leading to server proactively sending RST to idle keep-alive connections
- warningFirewall or security group with session timeouts that kill long-lived TCP connections
- warningTCP keepalive settings misconfiguration: kernel sends keepalive probes too late or not at all, causing stale connections
Concrete fix directions. Pick the one that matches your root cause.
- buildAlign keep-alive timeouts: set server `keepalive_timeout` (e.g., 65s) slightly higher than client idle timeout (e.g., 60s) to prevent premature closure.
- buildEnable TCP keepalive on the server: `sysctl -w net.ipv4.tcp_keepalive_time=300` and ensure application uses SO_KEEPALIVE.
- buildImplement graceful shutdown: catch SIGTERM and stop accepting new connections, drain existing keep-alive connections with a drain timeout.
- buildIncrease connection pool limits on the server and upstream to avoid RST due to pool exhaustion.
- buildFor proxies, set `timeout http-keep-alive` in HAProxy to be longer than client idle timeout, or disable keep-alive at proxy if not needed.
- buildAdd retry logic at the client: on ECONNRESET, reconnect and retry the request (idempotent methods only).
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, run `curl -v --keepalive-time 10 --max-time 120 http://your-server/` and confirm no RST after idle.
- verifiedMonitor `ss -tanpo` for CLOSE_WAIT sockets decreasing over time.
- verifiedUse `tcpdump` on the server interface to confirm TCP FIN/ACK exchange occurs instead of RST after idle timeout.
- verifiedLoad test with `wrk -t4 -c100 -d60s --latency http://your-server/` and check zero 'connection reset' errors in logs.
- verifiedCheck application health check endpoints: ensure they complete without RST.
- verifiedDeploy canary with fix and compare 'connection reset by peer' error rates in monitoring dashboards.
Things that make this bug worse or harder to find.
- warningSetting keepalive_timeout too high (e.g., >300s) can lead to resource exhaustion and increased memory usage.
- warningBlindly disabling keep-alive: some applications benefit from connection reuse; better to fix timeout alignment.
- warningIgnoring intermediate proxies: a reset might come from a load balancer, not the origin server.
- warningApplying kernel TCP keepalive changes without verifying application-level keepalive settings; they serve different purposes.
- warningAssuming all RSTs are harmful: some are legitimate (e.g., server rejecting a request). Distinguish by payload analysis.
- warningUsing `tcpkill` or similar tools without understanding the protocol — they can cause RSTs themselves.
The 60-second RST that took down payments
Timeline
- 09:15PagerDuty alert: payment service 502 error rate spikes to 12%
- 09:18Check ALB logs: 'connection reset by peer' from upstream, target group healthy
- 09:22SSH to Go service instance, run `ss -tanpo | grep CLOSE_WAIT` — 200+ sockets in CLOSE_WAIT
- 09:25tcpdump on service: see TCP RST from ALB IP exactly 60 seconds after last data on keep-alive connections
- 09:30Check HAProxy config: `timeout client 60s`, `timeout server 60s`, but Go service keep-alive timeout is 120s
- 09:35HAProxy idle timeout (60s) fires, sends RST to both sides, but Go service still considers connection alive
- 09:40Fix: set HAProxy `timeout client 65s` and `timeout server 65s`, aligned with Go's 120s? Actually adjust Go to 55s to stay under proxy
- 09:45Deploy config change, error rate drops to 0.1%
I was on-call when the payment service started throwing 502s. The immediate suspect was the upstream database, but DB metrics were fine. ALB logs showed 'connection reset by peer' from the Go service, yet the service health checks passed. I ran `ss -tanpo` on the Go instance and saw hundreds of sockets in CLOSE_WAIT — a classic sign of the peer closing the connection without the application noticing.
I captured traffic with tcpdump and filtered for RST packets. Every RST came from the ALB IP exactly 60 seconds after the last data transfer on a keep-alive connection. Our HAProxy configuration had `timeout client 60s` and `timeout server 60s`, but the Go service had its own keep-alive timeout set to 120 seconds. When HAProxy's idle timer expired, it sent a RST to both ends. The Go service, thinking the connection was still alive, later tried to write and got a broken pipe.
The fix was simple: align timeouts. We reduced the Go service's keep-alive timeout to 55 seconds (just under HAProxy's 60s) so the Go server would gracefully close idle connections before HAProxy killed them. We also added a small buffer to HAProxy's timeouts (65s) to avoid edge cases. After the change, the 502s vanished. The lesson: always know every hop's timeout values, and ensure the one closest to the client closes first.
Root cause
Timeout mismatch: HAProxy's idle timeout (60s) was shorter than the Go service's keep-alive timeout (120s), causing HAProxy to send RST on idle connections while the Go service still considered them reusable.
The fix
Set the Go service's keep-alive timeout to 55s (below HAProxy's 60s) and increased HAProxy client/server timeouts to 65s to avoid premature RST.
The lesson
Map out the entire request path's timeout hierarchy. The server that closes idle connections first should be the one closest to the client, otherwise you get RSTs.
A normal TCP connection termination uses a FIN handshake: one side sends FIN, the other acknowledges, then the reverse. This is a graceful close that allows both sides to flush data. A RST (reset) is an abrupt termination — no more data is accepted, and the connection is destroyed immediately. In HTTP keep-alive, the client and server agree to reuse the TCP connection for multiple requests. If one side sends a RST while the other expects to reuse the connection, the next request fails with 'connection reset by peer'.
The root cause often lies in timeout mismatches. For example, a server may have a keepalive_timeout of 75 seconds, meaning it will close idle connections after 75 seconds of inactivity. If a client (or proxy) has a longer idle timeout, it may try to send a request after the server has already closed the connection. The server's close sends a FIN, but if the client's TCP stack hasn't processed the FIN yet (maybe due to buffering), the client sends a new request, and the server responds with a RST because the connection is already closing. Alternatively, the server may send a RST directly if it forcefully closes the socket (e.g., `SO_LINGER` with zero timeout).
To pinpoint the source of the RST, capture traffic on both sides of the connection. Run `tcpdump -i any -s0 port 80 -w capture.pcap` on the client and server simultaneously. In Wireshark, apply a filter `tcp.flags.reset == 1`. Look at the IP address that sent the RST — that's the one closing abruptly. Also check the sequence numbers: a RST with a sequence number matching the expected next sequence number indicates a legitimate reset; out-of-window resets can be spoofed or due to kernel craziness.
Pay attention to the time delta between the last data packet and the RST. If it matches a known timeout (e.g., exactly 60 seconds), you've found the misconfiguration. Also look for TCP keepalive probes (packets with no payload and sequence number equal to last ack minus one) preceding the RST — that means the kernel's keepalive triggered, not the application's timeout.
A non-obvious cause is when a server's connection pool is exhausted. For example, a Go HTTP server with a maximum of 1000 concurrent connections might reject new connections by sending RST when the pool is full. This can happen even on keep-alive connections if the server is under heavy load and decides to proactively close idle connections to free up resources. The server may set a `ConnContext` with a deadline or use `http.Transport.MaxIdleConnsPerHost` to limit idle connections, leading to RST on excess idle sockets.
To detect this, monitor the server's open file descriptors (`lsof -p <pid> | wc -l`) and compare with connection limits. In Go, use `runtime.NumGoroutine()` and `net/http/pprof` to see connection states. Also check server logs for messages like 'too many open files' or 'connection reset by peer' with a high frequency during traffic spikes.
Stateful firewalls (e.g., iptables conntrack, AWS Security Groups, ALB) track TCP connection states. If a connection is idle longer than the firewall's session timeout, the firewall may forget the connection and drop subsequent packets, causing the client to see a timeout or RST. AWS ALB has a default idle timeout of 60 seconds for HTTP connections, which can be increased up to 4000 seconds. Many cloud load balancers also have a maximum keep-alive timeout that overrides the application's setting.
The fix involves aligning all timeouts in the path: client -> CDN -> load balancer -> reverse proxy -> application server. Use the shortest timeout as the effective keep-alive timeout. Tools like `tracepath` or `mtr` can help map the network path, but you'll need to check each component's documentation for timeout settings. A common pattern is to set the application's keep-alive timeout to be slightly lower than the load balancer's (e.g., 55s vs 60s) so the application closes idle connections gracefully before the load balancer does.
Frequently asked questions
What is the difference between 'connection reset by peer' and 'connection timed out'?
'Connection reset by peer' means the remote side sent a TCP RST packet, indicating an abrupt closure. 'Connection timed out' means the client didn't receive any response within the timeout period (no RST, no data). The reset is a faster failure, often due to misconfiguration or server crash, while timeout suggests network loss or server unresponsiveness.
Can a client cause a keep-alive connection reset?
Yes. If a client closes a keep-alive connection abruptly (e.g., by calling `close()` without a graceful shutdown, or by using `SO_LINGER` with zero timeout), it sends a RST. The server then sees 'connection reset by peer' when trying to read the next request. This can happen with buggy HTTP clients or when a client crashes.
How do I differentiate between a reset from a proxy vs the origin server?
Use tcpdump on the client side to capture the RST. Look at the source IP of the RST packet: if it's the proxy's IP, the proxy reset the connection; if it's the origin server's IP, the origin reset it. Also check the TTL: if the TTL decrements by one compared to previous packets, it's likely from the next hop.
Should I disable keep-alive to fix connection reset errors?
Only as a last resort. Keep-alive reduces latency and server load by reusing connections. Disabling it will increase connection overhead and may cause other issues. Instead, fix the timeout alignment or handle graceful shutdown properly. If you must disable it, do so at the proxy level to minimize impact.
What is the role of TCP keepalive (SO_KEEPALIVE) in connection resets?
TCP keepalive is a kernel-level mechanism to detect dead peers by sending periodic probes. If no response is received after a number of probes, the kernel sends a RST. This can cause 'connection reset by peer' if the peer is actually alive but the keepalive probes are not acknowledged due to network issues. Proper tuning of `tcp_keepalive_time`, `tcp_keepalive_intvl`, and `tcp_keepalive_probes` can prevent false positives.