What this usually means
A connection reset during the TLS handshake—specifically after the TCP handshake completes but before the server sends its ServerHello—usually indicates that the server process crashed or deliberately closed the connection after receiving the ClientHello. This is not a TCP-level issue; it's an application-layer failure. Common causes include: the server lacking a valid certificate chain, a cipher mismatch where the server cannot negotiate any common cipher, SNI misconfiguration causing the server to reject the request, or an internal SSL library error that forces the server to abort. The RST packet is the TCP layer's way of signaling that the application closed the socket without completing the TLS handshake.
The first ten minutes — establish facts before touching code.
- 1Run 'openssl s_client -connect host:port -debug' to see exactly where the handshake fails; look for 'R' in the output indicating reset.
- 2Capture traffic with tcpdump: 'tcpdump -i any -w tls.pcap host target_host and port 443'; analyze with Wireshark, filter 'tls.handshake.type == 1' and look for RST after ClientHello.
- 3Check server logs for SSL error entries: 'grep -i ssl /var/log/nginx/error.log' or 'journalctl -u nginx | grep -i ssl'.
- 4Verify certificate chain with 'openssl s_client -showcerts -servername example.com -connect host:port' and ensure no 'unable to get local issuer certificate' errors.
- 5Test with different TLS versions: 'openssl s_client -tls1_2 -connect host:port' then '-tls1_3' to isolate version-specific issues.
The specific files, logs, configs, and dashboards that usually own this bug.
- search/var/log/nginx/error.log (or apache2/error.log) - search for 'SSL' or 'handshake failed'
- search/var/log/syslog or /var/log/messages - look for kernel messages about connection resets
- searchApplication logs (e.g., /var/log/myapp/error.log) - check for SSL library exceptions
- searchLoad balancer logs (HAProxy: /var/log/haproxy.log, Nginx: /var/log/nginx/access.log with SSL info)
- searchCertificate files: /etc/ssl/certs/ and /etc/ssl/private/ - verify permissions and format
- searchWireshark/tcpdump capture - filter 'tls.handshake.type' and 'tcp.flags.reset == 1'
- searchOpenSSL s_client output - look for 'R' in the debug output or 'SSL_R_*' errors
Practical causes, not theory. These are the things you will actually find.
- warningServer certificate file is missing, expired, or does not match the private key (e.g., key file is 0 bytes)
- warningCipher suite mismatch: server has no ciphers compatible with client's ClientHello (e.g., server only supports old ciphers, client only TLS 1.3)
- warningSNI misconfiguration: server fails to select a certificate for the requested hostname and resets connection
- warningServer process crashes due to SSL library bug or memory exhaustion when processing ClientHello
- warningReverse proxy or load balancer terminates TLS but has incorrect backend TLS settings, causing upstream reset
- warningFirewall or IDS inspects TLS handshake and sends RST if it detects invalid parameters or protocol violations
Concrete fix directions. Pick the one that matches your root cause.
- buildRegenerate and install correct certificate chain: ensure fullchain includes intermediates and private key matches
- buildUpdate server cipher configuration: enable a broader set (e.g., 'ECDHE-RSA-AES128-GCM-SHA256') and order by strength
- buildConfigure SNI properly: add 'ssl_certificate' and 'ssl_certificate_key' for each server_name in Nginx
- buildRestart the SSL library or upgrade to a patched version (e.g., OpenSSL 1.1.1t for CVE fixes)
- buildIncrease server resources: check ulimit -n, memory, and connection backlog; add 'ssl_early_data off' if using TLS 1.3
- buildAdd firewall exemption for expected TLS traffic, or disable deep packet inspection for the port
A fix you cannot prove is a guess. Close the loop.
- verifiedRun 'openssl s_client -connect host:port -servername hostname < /dev/null' and confirm it outputs 'CONNECTED(00000003)' and shows server certificate
- verifiedUse 'curl -v https://hostname/' and verify no 'Connection reset by peer' error
- verifiedMonitor server logs after fix: 'tail -f /var/log/nginx/error.log | grep -i ssl' should show no handshake errors
- verifiedLoad test with 'ab -n 1000 -c 10 https://hostname/' and check for zero SSL errors
- verifiedVerify from multiple clients with different TLS libraries (e.g., wget, Python requests, Java HttpURLConnection)
Things that make this bug worse or harder to find.
- warningAssuming it's a firewall issue without first checking server-side logs and OpenSSL s_client output
- warningRebooting the server without capturing the TLS handshake packets first (ephemeral evidence lost)
- warningChanging cipher suites randomly without understanding the server's supported list (use 'openssl ciphers -v')
- warningIgnoring certificate chain completeness: missing intermediates cause resets in some clients
- warningDisabling TLS 1.3 completely to work around the problem, which reduces security and performance
- warningPatching the load balancer without checking the backend TLS configuration
Production Outage: TLS Handshake Failure on Payment Gateway
Timeline
- 13:45PagerDuty alerts: payment gateway returns 502 for 50% of requests
- 13:47Check health endpoint: 'curl -v https://payments.internal.example.com/health' returns 'Connection reset by peer'
- 13:50Run openssl s_client: handshake fails after ClientHello with 'SSL_ERROR_SYSCALL'
- 13:52Capture tcpdump: see RST after ServerHello? No, RST immediately after ClientHello
- 13:55Check Nginx error log: 'SSL_do_handshake() failed (SSL: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher)
- 14:00Check Nginx ssl_ciphers: 'HIGH:!aNULL:!MD5' - seems fine. Check Go app's supported ciphers: uses TLS 1.3 only
- 14:05Realize Nginx reverse proxy negotiates TLS with client, but backend Go app only supports TLS 1.3. Nginx tries to forward TLS 1.2? No, Nginx terminates TLS, but the error is on the client-facing side.
- 14:10Double-check: the error is from client-facing Nginx. Why 'no shared cipher'? Check ssl_protocols: 'TLSv1.2 TLSv1.3' - should be fine. Check ssl_ciphers again: missing 'TLS_AES_128_GCM_SHA256' for TLS 1.3
- 14:15Fix: add 'TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384' to ssl_ciphers. Reload Nginx.
- 14:16Test openssl s_client: handshake succeeds. curl works. Alerts clear.
The pager woke me at 1:45 PM. Payment gateway was intermittently returning 502 errors. I immediately ssh'd into the Nginx box and ran curl against the internal endpoint. Connection reset by peer. That's not a typical 502—this was a TLS failure. I grabbed a coffee and opened three terminals: one for openssl s_client, one for tcpdump, and one for logs.
The s_client output showed the connection reset right after ClientHello, no ServerHello. The tcpdump confirmed: SYN, SYN-ACK, ACK, then ClientHello, then RST. So TCP was fine, but TLS died. I grepped the Nginx error log and found the smoking gun: 'no shared cipher'. I thought, 'That's odd, we have a broad cipher set.' I checked the protocols: TLSv1.2 and TLSv1.3 were both enabled. Then I looked more carefully at the cipher list: it only contained TLS 1.2 ciphers. The TLS 1.3 ciphers were not explicitly listed, and OpenSSL 1.1.1f requires them to be included. Our clients were using TLS 1.3, so the handshake failed.
The fix was simple: add the TLS 1.3 cipher suites to the ssl_ciphers directive. After reloading Nginx, the handshake succeeded. The incident lasted 30 minutes because I initially overlooked the TLS 1.3 cipher requirement, assuming they were automatically included. Lesson learned: always verify that your cipher list covers all enabled protocol versions.
Root cause
Nginx ssl_ciphers directive did not include TLS 1.3 cipher suites (TLS_AES_128_GCM_SHA256, etc.), causing 'no shared cipher' error for TLS 1.3 clients, which led to connection reset.
The fix
Added 'TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384' to the ssl_ciphers directive in Nginx configuration and reloaded.
The lesson
When enabling TLS 1.3, you must explicitly include TLS 1.3 cipher suites in the ssl_ciphers directive; they are not automatically inherited. Always test handshake with openssl s_client using specific protocol versions.
When you see a connection reset during TLS handshake, the first step is to capture the packets and filter for the handshake sequence. Use tcpdump: 'tcpdump -i any -s 0 -w capture.pcap host <server_ip> and port 443'. Then open in Wireshark. Filter by 'tls.handshake.type == 1' to see ClientHello. Then look for a TCP RST shortly after. If the RST comes before ServerHello, it's a server-side abort. If after ServerHello, the client may have rejected the certificate.
Examine the ClientHello: what cipher suites does the client offer? What SNI extension? Compare with server's configured ciphers. In Wireshark, follow TLS stream. If you see a TLS Alert before RST, that's a clue. Common alerts: '40' (handshake failure), '20' (bad certificate), '21' (decryption failed). The RST is often the TCP layer cleaning up after a library call to SSL_shutdown or close() without completing handshake.
'openssl s_client -connect host:port -debug' prints all handshake bytes. Look for 'R' in the output indicating read error. Add '-tlsextdebug' to see SNI. Add '-msg' to see handshake messages. If the connection resets, you'll see 'SSL_connect: error in SSLv2/v3 read server hello A'. The '-state' flag shows state machine transitions.
To test specific TLS versions: '-tls1_2', '-tls1_3'. For cipher suites: '-cipher <cipherlist>'. For SNI: '-servername <name>'. For certificate validation: '-verify_return_error' and '-CAfile <ca.pem>'. A successful handshake ends with 'SSL handshake has read N bytes and written M bytes'. A failure often prints 'SSL routines:ssl3_read_bytes:tlsv1 alert internal error' followed by a reset.
In Nginx, the 'ssl_ciphers' directive must include TLS 1.3 ciphers when 'ssl_protocols TLSv1.3' is set. Example: 'ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384;'. Missing these causes 'no shared cipher' for TLS 1.3 clients. Also ensure 'ssl_certificate' points to a full chain file containing intermediate certificates, not just the leaf cert.
Apache uses 'SSLCipherSuite' and 'SSLProtocol'. A common mistake is setting 'SSLCipherSuite HIGH:!aNULL:!MD5' which defaults to TLS 1.2 ciphers only. To include TLS 1.3, add 'TLS_AES_128_GCM_SHA256' explicitly. Also check 'SSLCertificateChainFile' or include intermediates in the certificate file. Missing intermediates cause 'sslv3 alert certificate unknown' in some clients.
When a load balancer (HAProxy, Nginx, AWS ALB) terminates TLS, it may fail if its cipher list doesn't match client offerings. HAProxy's 'ssl-default-bind-ciphers' must include TLS 1.3 ciphers. For ALB, you can only select predefined security policies; ensure the policy includes TLS 1.3. If the load balancer forwards traffic to a backend that also uses TLS, the backend handshake may also reset.
A tricky scenario: the load balancer terminates TLS, but the backend expects a different SNI hostname. The load balancer forwards the original SNI or sets a custom one. If there's a mismatch, the backend may reject the connection. Use 'openssl s_client -connect backend:port -servername expected_name' to test.
The client sends a list of cipher suites in order of preference. The server picks the first one from its own list that matches. If no match, the server sends a handshake failure alert and closes the connection, often resulting in RST. To debug, use 'openssl ciphers -v' on both client and server to see supported suites. Common mismatches: client only supports TLS 1.3 ciphers, server only has TLS 1.2 ciphers, or vice versa.
Another issue: elliptic curve support. If the client only supports certain curves (e.g., X25519) and the server doesn't, the handshake may fail. Check with 'openssl s_client -curves X25519'. Also, key exchange algorithm: if server requires DHE but client doesn't support it. The 'openssl s_client -cipher' option can force specific suites to isolate the problem.
Frequently asked questions
Why does TLS handshake fail with 'connection reset' but TCP handshake succeeds?
TCP handshake (SYN, SYN-ACK, ACK) establishes the transport layer connection. The TLS handshake runs on top of TCP. If the server process crashes or explicitly closes the socket upon receiving the ClientHello (e.g., due to cipher mismatch, missing certificate, or internal SSL error), TCP sends an RST to the client. The client sees 'connection reset by peer'. This indicates the server rejected the TLS negotiation, not a network problem.
How can I tell if the reset is from the server or a firewall?
Use packet capture. In Wireshark, look for the TCP source of the RST. If it's from the server IP, it's the server. If from a different IP (e.g., firewall/gateway), it's a network device. Also, firewalls often send RST with a TCP window of zero and may not have the TLS payload. Server resets usually occur after the ClientHello is fully received. Additionally, check if the reset is immediate after SYN-ACK (before ClientHello) — that's likely a firewall blocking the port.
What does 'no shared cipher' mean in Nginx error log?
It means that during the TLS handshake, the server (Nginx) could not find any cipher suite that both the client and server support. This happens when the client's list of cipher suites does not intersect with the server's ssl_ciphers list. Common causes: server only has TLS 1.2 ciphers but client only offers TLS 1.3 ciphers, or vice versa. Also, server might have disabled certain key exchange methods (e.g., DHE) that the client requires. Fix: ensure ssl_ciphers includes suites for all enabled protocol versions.
Can a missing intermediate certificate cause a connection reset?
Yes, but not directly. A missing intermediate certificate usually causes the client to fail certificate validation, which results in a TLS alert (e.g., 'certificate unknown') but the server does not reset. However, some clients or libraries may abort the connection with an RST if they cannot build a trusted chain. More commonly, a missing intermediate leads to handshake failure visible as an alert, not a reset. Resets are more often from server-side issues like crashes or misconfigurations.
Why does curl work but openssl s_client fails?
curl and openssl s_client may use different TLS versions or cipher suites by default. curl often uses the system's TLS library (e.g., NSS, GnuTLS) while s_client uses OpenSSL. Also, curl may have a different CA bundle. To isolate, use 'curl --tlsv1.2' or 'openssl s_client -tls1_2'. If curl succeeds and s_client fails, it's likely a cipher or version mismatch. Compare the ClientHello from each tool using '-V' in s_client or '--verbose' in curl.