What this usually means
The client does not trust the certificate presented by the server because the certificate's issuer (the CA) is not in the client's trust store. With a self-signed certificate, the issuer is the certificate itself, so it must be explicitly added to the trust store or the client must be configured to skip verification (not recommended for production). The error can also occur if the certificate chain is incomplete (missing intermediate CAs) or if the certificate has expired or is used for the wrong hostname.
The first ten minutes — establish facts before touching code.
- 1Run 'openssl s_client -connect host:port -showcerts' to view the certificate chain and check for self-signed root.
- 2Check the certificate validity: 'openssl s_client -connect host:port 2>&1 | openssl x509 -noout -dates'.
- 3Verify the hostname matches: compare the certificate's Subject Alternative Names (SANs) with the host used in the client.
- 4Test with curl and the '-k' flag: if it works, the error is definitely a trust issue.
- 5Inspect application trust store: for Java, check 'cacerts' with 'keytool -list -keystore $JAVA_HOME/lib/security/cacerts'.
The specific files, logs, configs, and dashboards that usually own this bug.
- search/etc/ssl/certs/ — system CA certificates store (on Linux)
- search$JAVA_HOME/lib/security/cacerts — Java default trust store
- searchApplication configuration: environment variables like SSL_CERT_FILE, REQUESTS_CA_BUNDLE, or NODE_EXTRA_CA_CERTS
- searchKubernetes: ConfigMap or Secret containing CA bundle mounted into pods
- searchDockerfile: any custom CA certificates added via 'COPY' or 'RUN update-ca-certificates'
- searchServer's TLS configuration: the certificate file and key paths in nginx, Apache, or other web servers
Practical causes, not theory. These are the things you will actually find.
- warningThe client's trust store does not contain the self-signed root certificate.
- warningThe server is presenting a self-signed certificate instead of a publicly signed one (e.g., in development or internal services).
- warningThe certificate chain is incomplete: the server omitted intermediate certificates, so the client cannot build a path to a trusted root.
- warningThe certificate has expired (check dates) or is not yet valid.
- warningHostname mismatch: the certificate's CN or SANs don't match the hostname used in the client request.
- warningThe application uses a custom trust store that overrides the system default, missing the self-signed CA.
Concrete fix directions. Pick the one that matches your root cause.
- buildAdd the self-signed CA certificate to the client's trust store. For system-wide: copy to '/usr/local/share/ca-certificates/' and run 'sudo update-ca-certificates'.
- buildFor Java applications: import the cert into cacerts: 'keytool -import -alias myca -keystore cacerts -file ca.crt'.
- buildFor Node.js: set 'NODE_EXTRA_CA_CERTS' environment variable pointing to the CA certificate file.
- buildFor Python requests: set 'REQUESTS_CA_BUNDLE' environment variable or use 'verify=/path/to/cabundle.pem'.
- buildServe a proper certificate chain: on the server, concatenate the server certificate followed by intermediate(s) and root (if not self-signed).
- buildTemporarily disable verification for development only (never in production) using curl -k or by setting environment variables like 'NODE_TLS_REJECT_UNAUTHORIZED=0'.
A fix you cannot prove is a guess. Close the loop.
- verifiedRun 'curl -v https://host:port' and confirm no SSL errors.
- verifiedUse 'openssl s_client -connect host:port -CAfile ca.crt' and check for 'Verify return code: 0 (ok)'.
- verifiedTest the application's endpoint with the same client that was failing (e.g., if it was a Java app, run the same Java code).
- verifiedCheck the trust store contents to ensure the certificate is present with the correct alias.
- verifiedFor Kubernetes: exec into the pod and run 'curl' against the service endpoint to verify the fix.
- verifiedRun a full integration test suite that includes HTTPS calls.
Things that make this bug worse or harder to find.
- warningDisabling certificate verification globally (e.g., setting 'NODE_TLS_REJECT_UNAUTHORIZED=0' in production).
- warningAdding the certificate to the wrong trust store (e.g., adding to Java's cacerts but the app uses a custom one).
- warningForgetting to restart the application or service after updating the trust store.
- warningImporting the server certificate instead of the CA certificate (the one that signed it).
- warningUsing an expired certificate: always check validity dates before importing.
- warningOverwriting the system trust store instead of appending the new CA certificate.
Internal API Fails with 'self-signed certificate in certificate chain' After Migration
Timeline
- 09:15Deploy new internal service 'orders-api' with a self-signed certificate for inter-service mTLS.
- 09:20Client service 'billing' fails to connect: 'x509: certificate signed by unknown authority'.
- 09:25Check logs: both services are in same namespace, but billing can't verify orders-api's cert.
- 09:30Run 'openssl s_client -connect orders-api:8443 -showcerts' from billing pod and see self-signed root.
- 09:35Find that billing's trust store (mounted ConfigMap) is missing the CA that signed orders-api cert.
- 09:40Update the ConfigMap with the CA certificate and restart billing pods.
- 09:45Test connection: still failing — 'certificate has expired'.
- 09:50Check orders-api cert: it was generated with a validity of 1 day and expired overnight.
- 09:55Regenerate the server certificate with 1 year validity and redeploy.
- 10:00Connection succeeds. Add monitoring for certificate expiry.
We migrated our internal services to use mutual TLS with self-signed certificates. The orders-api team generated a new certificate and deployed it. Almost immediately, billing service started failing to connect. The error was 'x509: certificate signed by unknown authority'. I jumped on it because this was blocking a critical payment flow.
I started by checking the certificate chain from the billing pod using openssl. It showed the self-signed root. Then I checked billing's trust store (a ConfigMap mounted as a volume) and found it only had the old CA certificate. I updated the ConfigMap with the new CA and restarted the pods. But the error persisted.
I re-ran openssl with -CAfile and got 'verify error: certificate has expired'. I checked the server certificate dates and found it was only valid for 1 day and had already expired. I regenerated it with a 1-year validity and redeployed. That fixed the issue. Now we have a certificate expiry monitor in place.
Root cause
Orders-api's self-signed certificate had expired (1-day validity) and billing's trust store was missing the correct CA certificate.
The fix
Regenerated the server certificate with a 1-year validity, updated the ConfigMap with the CA certificate, and restarted billing pods.
The lesson
Always check certificate expiry dates first (it's quick). Also, ensure trust stores are synchronized across all clients.
When a client connects to a TLS server, the server sends its certificate chain. The client tries to build a path from the server's certificate to a trusted root CA in its trust store. If the server certificate is self-signed, it is its own root — so the client must have that exact certificate in its trust store. Otherwise, the path building fails with 'self-signed certificate'.
The chain can also fail if intermediate certificates are missing. The server should send the full chain (excluding the root if the client already has it). Use 'openssl s_client -showcerts' to see what the server sends. If the chain is incomplete, configure the server to include intermediates.
Inside Docker containers, the system trust store is often minimal. If your application depends on system CA certificates, you must add your self-signed CA via the Dockerfile (e.g., COPY ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates). For Java apps, you need to import into cacerts inside the image.
In Kubernetes, trust stores are typically mounted as ConfigMaps or Secrets. Ensure all pods that need to communicate have the correct CA bundle mounted. Also, be aware that base images like alpine have no update-ca-certificates — you may need to copy certs directly to /etc/ssl/certs.
Environment variables like 'NODE_TLS_REJECT_UNAUTHORIZED=0', 'CURL_CA_BUNDLE=/dev/null', or 'GODEBUG=x509ignoreCN=0' can disable verification entirely. This is a common quick fix but extremely dangerous in production because it opens the door to man-in-the-middle attacks.
If you must bypass for debugging, limit it to a single request. For example, 'curl -k https://host' bypasses verification for that one call. Never set environment variables globally in production.
To verify that a client trusts a server's certificate, use: 'openssl s_client -connect host:port -CAfile /path/to/ca.crt'. Look for 'Verify return code: 0 (ok)'. If it returns a non-zero code, the error message explains why (e.g., 19: self-signed certificate, 10: certificate has expired).
To test without trust: 'openssl s_client -connect host:port' and ignore the verify error. This is useful to see the certificate details even if trust fails.
Frequently asked questions
What is the difference between 'self-signed certificate' and 'self-signed certificate in certificate chain'?
'Self-signed certificate' means the server directly sent a self-signed cert that is not trusted. 'Self-signed certificate in certificate chain' means the chain includes a self-signed root that the client doesn't trust, even if the server cert itself is signed by an intermediate. The fix is the same: add the self-signed root to the trust store.
How do I add a self-signed certificate to the Java trust store?
Use keytool: 'keytool -import -alias myalias -keystore $JAVA_HOME/lib/security/cacerts -file ca.crt'. Default password is 'changeit'. For custom trust stores, specify the path with '-Djavax.net.ssl.trustStore'.
Why does my browser show 'NET::ERR_CERT_AUTHORITY_INVALID' even after importing the certificate?
Browsers have their own trust stores separate from the OS. You must import the certificate into the browser's certificate manager (Settings > Privacy & Security > Certificates). Also, ensure the certificate has the correct Subject Alternative Name (SAN) for the hostname.
Can I use a self-signed certificate in production?
It's discouraged because clients must be manually configured to trust it, which is error-prone and reduces security. For internal services, use a private CA (e.g., with cert-manager on Kubernetes) that is deployed to all clients. For public services, use a trusted CA like Let's Encrypt.
What does 'unable to verify the first certificate' mean?
It means the client received the server certificate but could not find a trusted root to verify it. This is typical of self-signed certificates or missing intermediates. Check the chain with openssl and ensure the CA is in your trust store.