What this usually means
Linkerd mTLS failures typically stem from one of three categories: (1) certificate rotation or expiration causing the proxy to reject the peer certificate, (2) network policy or sidecar injection issues that prevent the proxy from establishing the control-plane connection needed for trust bundle distribution, or (3) application-level policy (Server/ServerAuthorization resources) blocking the handshake. The root cause is almost never a broken mTLS implementation in Linkerd itself—it's almost always a configuration or operational gap in the cluster.
The first ten minutes — establish facts before touching code.
- 1Run `linkerd viz check --proxy` to surface any proxy-level health issues
- 2Inspect proxy logs: `kubectl logs <pod> -c linkerd-proxy | grep -i 'tls\|handshake\|certificate'`
- 3Check identity controller logs: `kubectl logs -n linkerd deploy/linkerd-identity -c identity -f | grep -i 'error\|expire\|rotate'`
- 4Verify trust anchors: `kubectl -n linkerd get configmap linkerd-identity-trust-roots -o yaml | openssl x509 -noout -text | grep -A2 'Validity'`
- 5Review Server/ServerAuthorization resources: `kubectl get serverauthorization -A -o wide` for any blocking rules
The specific files, logs, configs, and dashboards that usually own this bug.
- searchProxy logs per pod: `kubectl logs <pod> -c linkerd-proxy`
- searchIdentity controller logs: `kubectl logs -n linkerd deploy/linkerd-identity -c identity`
- searchLinkerd viz metrics: `linkerd viz top -n <namespace> -o wide`
- searchConfigMap `linkerd-identity-trust-roots` in namespace linkerd
- searchDestination controller logs: `kubectl logs -n linkerd deploy/linkerd-destination -c destination`
- searchNetwork policy definitions that may affect port 443 or 8443
- searchPod annotations for sidecar injection: `kubectl describe pod <pod> | grep -A5 Annotations`
Practical causes, not theory. These are the things you will actually find.
- warningTrust anchor certificate rotation was not performed correctly, causing mismatched roots
- warningIdentity controller has stale or expired issuer credentials after control-plane upgrade
- warningNetworkPolicy blocks egress to the identity controller on port 443
- warningSidecar injection skipped for some pods due to missing `linkerd.io/inject: enabled` annotation
- warningServer/ServerAuthorization resource misconfigured with wrong service account or namespace selector
- warningKubernetes Secret `linkerd-identity-issuer` missing or containing expired cert
- warningProxy version mismatch between data plane and control plane (e.g., after partial upgrade)
Concrete fix directions. Pick the one that matches your root cause.
- buildRotate trust anchor correctly: `step certificate create root.linkerd.cluster.local ca.crt ca.key --profile root-ca --not-after=87600h` then `linkerd upgrade --identity-trust-anchors-file=ca.crt`
- buildRestart identity controller after certificate update: `kubectl rollout restart -n linkerd deploy/linkerd-identity`
- buildEnsure all pods have sidecar injected: patch deployment with `linkerd.io/inject: enabled` annotation
- buildAdd network policy egress rule allowing pods to reach identity controller (port 443) and destination controller (port 8086)
- buildRemove or correct overly restrictive ServerAuthorization: `kubectl delete serverauthorization -n <ns> <name>`
- buildUpgrade all data plane proxies to match control plane version: `linkerd upgrade --proxy-version=<version>`
A fix you cannot prove is a guess. Close the loop.
- verifiedRe-run `linkerd viz check --proxy` and confirm all checks pass
- verifiedSend test HTTP request between services: `kubectl exec deploy/sleep -- /bin/sh -c 'curl -s http://hello:8080'` and confirm 200
- verifiedCheck `linkerd viz edges` and verify all edges show 'yes' under mTLS
- verifiedMonitor proxy logs for absence of TLS errors over a 5-minute window
- verifiedUse `linkerd viz stat -n <ns> deploy/<svc> -o wide` to confirm success rate is 100%
- verifiedVerify certificate expiry: `kubectl -n linkerd get secret linkerd-identity-issuer -o yaml | openssl x509 -noout -enddate`
Things that make this bug worse or harder to find.
- warningDo not delete the identity Secret manually—this breaks the entire mesh
- warningAvoid mixing self-signed certificates with the default trust anchor; always use the same root CA
- warningDon't upgrade control plane without also running `linkerd upgrade` on data plane proxies
- warningNever set `identity.disabled: true` in the mesh config—it disables mTLS entirely
- warningDon't ignore proxy logs because they are verbose; filter with grep for errors
- warningAvoid using wildcard network policies that block egress to Linkerd control plane pods
Identity Controller Crash Loops After Certificate Rotation
Timeline
- 14:00Team rotated trust anchor using `linkerd upgrade --identity-trust-anchors-file=new-ca.crt`
- 14:02Most services show TLS handshake failures in logs; 503s spike
- 14:05Identity controller enters CrashLoopBackOff with 'issuer certificate expired' error
- 14:10`kubectl logs -n linkerd deploy/linkerd-identity -c identity` shows 'x509: certificate has expired or is not yet valid'
- 14:15Check `linkerd-identity-issuer` Secret—its certificate is still the old one
- 14:20Realize the upgrade command replaced trust roots but did not rotate the issuer certificate itself
- 14:25Generate new issuer certificate signed by the new root CA using `linkerd identity issue`
- 14:30Restart identity controller and all proxy pods; mTLS recovers
- 14:35Confirm with `linkerd viz check --proxy` all green
The incident started after a scheduled trust anchor rotation. We followed the documented procedure: generated a new root CA with step CLI and ran `linkerd upgrade` with the new trust anchors file. The command succeeded without errors, but within minutes services started returning 503s. Proxy logs showed TLS handshake failures with 'certificate verify failed'.
I immediately checked the identity controller logs and found it crashing with 'issuer certificate expired'. The issuer certificate inside the `linkerd-identity-issuer` Secret was still signed by the old root CA, which was no longer trusted. The upgrade replaced the trust bundle but did not automatically re-issue the issuer certificate.
We generated a new issuer certificate using `linkerd identity issue --issuer-cert-file new-issuer.crt ...` and updated the Secret. After restarting the identity controller and performing a rolling restart of all injected deployments, mTLS handshakes succeeded again. The fix took about 30 minutes, but it was entirely avoidable if we had issued a new issuer certificate during the rotation.
Root cause
Trust anchor rotation without regenerating the identity controller's issuer certificate, causing the issuer to be signed by an untrusted root.
The fix
Generate a new issuer certificate signed by the new root CA, update the Secret, and restart identity controller and all proxies.
The lesson
Always rotate both the trust anchor and the issuer certificate together. Use `linkerd upgrade` with `--identity-issuer-certificate-file` flag to do both in one step.
Linkerd proxies (linkerd-proxy) establish mutual TLS connections using certificates issued by the identity controller. Every proxy has a private key and a certificate signed by the identity controller's issuer CA. The trust bundle (root CA) is distributed to all proxies via the control plane.
The handshake works as follows: Proxy A connects to Proxy B. Proxy A presents its certificate signed by the issuer. Proxy B verifies the certificate using the trust bundle. If the issuer's certificate is not in the trust bundle or has expired, the handshake fails with 'certificate verify failed' or 'unknown authority'.
The most common cause of mTLS failures is expired certificates. Check the identity controller's issuer certificate: `kubectl -n linkerd get secret linkerd-identity-issuer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text | grep -A2 Validity`. Compare the 'Not Before' and 'Not After' dates.
If the issuer certificate is expired, generate a new one: `linkerd identity issue --issuer-cert-file issuer.crt --issuer-key-file issuer.key — expiry=87600h` and update the Secret. Then restart the identity controller and all proxy pods.
Also verify the trust anchor certificate: `kubectl -n linkerd get configmap linkerd-identity-trust-roots -o jsonpath='{.data.ca-bundle\.crt}' | openssl x509 -noout -text | grep -A2 Validity`. If it's expired, rotate it using `linkerd upgrade --identity-trust-anchors-file=new-ca.crt`.
Pods need to communicate with the identity controller (port 443) and destination controller (port 8086) to receive certificates and service discovery. If a NetworkPolicy blocks egress to these endpoints, the proxy cannot obtain its identity certificate.
To test, exec into a pod and try to connect: `kubectl exec deploy/myapp -- /bin/sh -c 'curl -sk https://linkerd-identity.linkerd.svc.cluster.local:443/health'`. If it fails, check NetworkPolicy in the pod's namespace. Ensure there is an egress rule allowing TCP to the linkerd namespace on ports 443 and 8086.
Example NetworkPolicy fix: `kubectl apply -f - <<EOF apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-linkerd-egress namespace: myapp spec: podSelector: {} egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: linkerd ports: - port: 443 protocol: TCP - port: 8086 protocol: TCP policyTypes: - Egress EOF`
Linkerd proxies are compatible only within the same minor version. If you upgrade the control plane (e.g., from 2.12.0 to 2.12.1) but leave some proxy versions at 2.11.x, the handshake may fail due to protocol changes or certificate format differences.
Check proxy versions: `kubectl get pods -A -o custom-columns=NAMESPACE:.metadata.namespace,POD:.metadata.NAME,PROXY_VERSION:.metadata.labels.linkerd\.io/proxy-version`. Any version not matching the control plane version (e.g., from `linkerd version`) indicates a mismatch.
Fix by running `linkerd upgrade --proxy-version=<version>` and restarting the pods. The safest approach is to use `linkerd upgrade` without the proxy-version flag, which upgrades both control and data plane to the latest stable.
Frequently asked questions
How do I check if mTLS is actually enabled between two services?
Use `linkerd viz edges -n <namespace>` and look for the 'mTLS' column. If it says 'yes', mTLS is active. For a specific pair, run `linkerd viz stat deploy/<svc1> -n <ns>` and check the 'success rate' and 'TLS' metrics.
What port does Linkerd use for mTLS?
Linkerd intercepts traffic on port 4143 for inbound and port 4140 for outbound by default. The actual mTLS handshake happens over the proxy's inbound listener. You don't need to configure these ports unless you're writing network policies.
Why do I see 'certificate verify failed' even though the certificate is valid?
This usually means the issuer certificate is not trusted by the peer. Check the trust bundle in ConfigMap `linkerd-identity-trust-roots`—it must contain the root CA that signed the issuer certificate. Also ensure the issuer certificate has not expired.
Can I disable mTLS temporarily for debugging?
Yes, but it's not recommended. You can set the annotation `config.linkerd.io/skip-outbound-ports: "*"` on a deployment to skip mTLS for all outbound traffic. Alternatively, modify the ServerAuthorization to allow unauthenticated traffic as a last resort.
How do I rotate the identity issuer certificate without downtime?
Generate a new issuer certificate and update the Secret. Then restart the identity controller. Proxies will automatically fetch the new certificate on next renewal (default 24h). To force immediate renewal, restart all proxy pods with `kubectl rollout restart`.