What this usually means
When Envoy returns 503, it means the upstream cluster has no healthy endpoints or connections are being reset. In Istio, this often stems from mismatched service ports, missing or misconfigured ServiceEntry, TLS handshake failures due to mTLS mode mismatch (e.g., STRICT vs PERMISSIVE), or Envoy's outlier ejection removing all endpoints. It can also be a transient issue if the upstream pod is not ready or the endpoint controller hasn't updated.
The first ten minutes — establish facts before touching code.
- 1kubectl exec -it <pod> -c istio-proxy -- curl -s http://localhost:15000/clusters | grep <upstream-cluster> to check healthy endpoints
- 2kubectl logs <pod> -c istio-proxy --tail=100 | grep '503' to find recent 503 responses
- 3kubectl exec -it <pod> -c istio-proxy -- curl -s http://localhost:15000/listeners | grep <port> to confirm listener config
- 4kubectl get endpoints <service> -n <namespace> to verify endpoints exist
- 5kubectl exec -it <pod> -c istio-proxy -- curl -s http://localhost:15000/config_dump | jq '.configs[0].dynamicActiveClusters' to inspect cluster config
The specific files, logs, configs, and dashboards that usually own this bug.
- searchEnvoy admin endpoint: http://localhost:15000/clusters, /listeners, /config_dump
- searchistio-proxy container logs: kubectl logs <pod> -c istio-proxy
- searchPilot/istiod logs for xDS updates: kubectl logs -n istio-system <istiod-pod>
- searchKubernetes Endpoints resource: kubectl get endpoints <service> -n <namespace>
- searchDestinationRule and VirtualService YAML: kubectl get dr <name> -o yaml, kubectl get vs <name> -o yaml
- searchmTLS settings (PeerAuthentication) in the namespace: kubectl get peerauthentication -A
- searchApplication container logs for any upstream errors
Practical causes, not theory. These are the things you will actually find.
- warningService port mismatch: container port in Deployment differs from service targetPort
- warningmTLS STRICT mode but upstream pod lacks sidecar or uses PERMISSIVE
- warningOutlier ejection due to repeated failures, causing zero healthy endpoints
- warningMissing or incorrect ServiceEntry for external services
- warningEnvoy connection pool exhausted or circuit breakers tripped
- warningDestinationRule with inconsistent subsets or trafficPolicy (e.g., tls mode mismatch)
- warningEndpoint not ready because pod hasn't passed readiness probe
Concrete fix directions. Pick the one that matches your root cause.
- buildAlign service targetPort with container port in the Deployment
- buildSet exportTo: '*' in ServiceEntry for cross-namespace access
- buildFor mTLS mismatch, set PeerAuthentication to PERMISSIVE or update DestinationRule tls mode
- buildReset outlier ejection by adjusting outlierDetection settings in DestinationRule (e.g., consecutive_5xx_errors = 5)
- buildIncrease Envoy connection pool size via DestinationRule trafficPolicy.connectionPool
- buildRestart pilot/istiod to force xDS re-sync
- buildIf using headless service, ensure endpoints have correct port name
A fix you cannot prove is a guess. Close the loop.
- verifiedCheck Envoy clusters: healthy endpoints count > 0 and no 'degraded' or 'excluded' endpoints
- verifiedSend test requests through sidecar: kubectl exec <pod> -c istio-proxy -- curl -v http://<service>:<port>
- verifiedRun istioctl proxy-status <pod> to confirm sidecar is receiving xDS updates
- verifiedTail istio-proxy logs for successful upstream responses (200) after fix
- verifiedVerify mTLS: istioctl authn tls-check <pod> <upstream-service>.<namespace>.svc.cluster.local
- verifiedRoll back and re-apply changes, then monitor error rates
Things that make this bug worse or harder to find.
- warningBlindly restarting pods without checking Envoy admin first
- warningAssuming mTLS is the problem when it's actually a port mismatch
- warningModifying DestinationRule without verifying the VirtualService routing logic
- warningSetting outlier ejection too aggressively (e.g., consecutive_5xx_errors = 1)
- warningIgnoring readiness probe failures that cause endpoints to be removed
- warningEditing Envoy config directly via admin (it's ephemeral; use Istio resources)
Nightly Batch Job 503s After mTLS Switch to STRICT
Timeline
- 02:15PagerDuty alert: 503 error rate > 5% for payment-service
- 02:20Check payment-service pods: all Running, readiness passed
- 02:25Envoy clusters for payment-service show 0/5 healthy endpoints
- 02:30Check DestinationRule: tls mode set to ISTIO_MUTUAL, but upstream uses PERMISSIVE
- 02:35Check PeerAuthentication in namespace: STRICT, applied 10 minutes ago
- 02:40Revert PeerAuthentication to PERMISSIVE
- 02:45Endpoints become healthy, 503s drop to 0
- 02:50Update DestinationRule to match mTLS settings and re-apply STRICT
At 02:15, I got paged that payment-service was returning 503s. The on-call had just enabled mTLS STRICT across the namespace as part of a security hardening. I checked the pods: all running, readiness probes passing. But the error rate was climbing. I knew from past runbooks to check Envoy's cluster status.
I exec'd into a sidecar and hit /clusters. The upstream cluster for payment-service showed 0/5 healthy endpoints. That was weird because the pods were up. I then checked the DestinationRule and saw tls mode: ISTIO_MUTUAL. But the upstream service had its PeerAuthentication set to PERMISSIVE (the default). The mTLS mismatch caused Envoy to attempt TLS handshake that the upstream couldn't handle, so connections were reset.
I quickly reverted the PeerAuthentication to PERMISSIVE, and within minutes the healthy endpoints count returned to 5/5. The 503s stopped. After confirming the fix, I updated the DestinationRule to also use DISABLE for now and scheduled a proper mTLS rollout with coordinated changes across all services.
Root cause
mTLS mismatch: DestinationRule enforced ISTIO_MUTUAL while upstream PeerAuthentication was PERMISSIVE, causing TLS handshake failures and Envoy marking all endpoints unhealthy.
The fix
Reverted PeerAuthentication to PERMISSIVE and aligned DestinationRule tls mode to match, then re-applied STRICT after coordinated updates.
The lesson
Always verify mTLS settings across both DestinationRule and PeerAuthentication before enabling STRICT mode. Use istioctl authn tls-check to validate end-to-end.
Envoy tracks each upstream endpoint in a cluster with states: HEALTHY, DEGRADED, EXCLUDED, or DRAINING. When you see 503 and 0/5 healthy, it means all endpoints are in non-healthy states. Check /clusters for fields like 'healthy_status::healthy' or 'healthy_status::unhealthy'. Common reasons for unhealthy: active health check failures, outlier ejection, or connection errors (e.g., TLS).
Use `curl http://localhost:15000/clusters?format=json` to get structured data. Focus on the 'healthFlags' field. 'H' means healthy, 'U' unhealthy. If all are 'U', check the 'eds_health_score' and 'outlier_detection::*' counters. Outlier ejection counters show how many times the endpoint was ejected.
Istio supports STRICT, PERMISSIVE, and DISABLE mTLS modes. STRICT requires both sides to present certificates. If the upstream sidecar is in PERMISSIVE (accepts both TLS and plaintext), but the downstream sends TLS, the upstream may still accept. However, if downstream DestinationRule sets tls mode to ISTIO_MUTUAL, Envoy will initiate a TLS handshake. If the upstream doesn't have a sidecar or has mTLS DISABLE, the handshake fails, resulting in 'upstream_reset_before_response_started' and 503.
To debug, run `istioctl authn tls-check <pod> <upstream-service>.<namespace>.svc.cluster.local`. It shows the actual TLS mode used. Also check the upstream's PeerAuthentication and DestinationRule. A common fix is to set the DestinationRule's tls mode to DISABLE if the upstream doesn't support mTLS, or ensure both sides have PeerAuthentication set to STRICT.
Envoy's outlier detection can eject endpoints after consecutive 5xx errors. If your service has transient failures, Envoy may eject all endpoints, causing 503s. Check the 'outlier_detection::*' fields in /clusters. 'outlier_detection::success_rate' shows the success rate; if below threshold, endpoints are ejected. 'outlier_detection::ejections_consecutive_5xx' shows count.
To fix, adjust the DestinationRule's outlierDetection settings. For example, set 'consecutive_5xx_errors: 5' and 'interval: 30s'. You can also disable outlier detection temporarily by removing the config. Monitor the 'outlier_detection::ejections_active' to see if endpoints are being re-added after the interval.
A common trivial cause: the service's targetPort doesn't match the container's containerPort. When this happens, Envoy's endpoint controller sees no endpoints because the port doesn't match the pod's port. Check with `kubectl get endpoints <service>`: if the endpoints list has no port or wrong port, Envoy will treat them as unhealthy. Also check the 'eds_health_score' in clusters: if it's 0, the endpoint is not considered healthy.
Ensure the service spec has a targetPort that matches the container port. Also, if using named ports, the name must match between service and pod. Use `kubectl describe pod <pod>` to see the container ports. If the service is headless, the endpoints must have the correct port name as well.
Sometimes 503s occur because the sidecar hasn't received the latest configuration from Pilot (istiod). This can happen after a change if Pilot is slow or if there's a network partition. Check with `istioctl proxy-status <pod>`. It shows the sync status: 'SYNCED' or 'NOT_SYNCED'. If not synced, check Pilot logs for errors: `kubectl logs -n istio-system <istiod-pod>`.
A quick fix is to restart the sidecar by injecting a pod restart or using `istioctl proxy-reset <pod>`. However, persistent sync issues indicate a Pilot resource problem (e.g., too many services, memory pressure). Verify Pilot's memory and increase limits if needed.
Frequently asked questions
How do I check if Envoy is receiving the correct endpoints?
Run `kubectl exec <pod> -c istio-proxy -- curl -s http://localhost:15000/clusters | grep <upstream-cluster>`. Look for 'healthy_endpoints: <number>'. If 0, check the Kubernetes Endpoints resource and ensure port names match.
My service returns 503 intermittently. What could cause that?
Intermittent 503s often come from outlier ejection or circuit breaker trips. Check Envoy's clusters for 'outlier_detection::ejections_active' and 'circuit_breakers::cx_open'. Increase thresholds in DestinationRule's outlierDetection or connectionPool settings.
How do I enable verbose Envoy logging for 503 debugging?
Set Envoy log level to debug: `kubectl exec <pod> -c istio-proxy -- curl -X POST http://localhost:15000/logging?level=debug`. Then tail logs: `kubectl logs <pod> -c istio-proxy -f`. To revert: `level=warning`.
Can mTLS cause 503 even if both sides have sidecars?
Yes. If the DestinationRule sets tls mode to ISTIO_MUTUAL but the PeerAuthentication is DISABLE or PERMISSIVE and the upstream sidecar doesn't present a certificate, the handshake fails. Always use `istioctl authn tls-check` to verify the negotiated mode.
What does 'upstream_reset_before_response_started' mean?
It means Envoy established a connection to the upstream but the upstream reset it before sending a response. Common causes: TLS handshake failure, upstream timeout, or the upstream crashing. Check the upstream pod logs and Envoy's connection logs.