Deployment reachability checklist

The symptoms

• •Continuous Integration reports a successful build and image push, yet external probes return 502 Bad Gateway responses with no application payload in the body.
• •Browser or API client receives a 503 Service Unavailable from a load balancer or ingress controller while internal cluster DNS resolves the service name.
• •curl or HTTP client times out connecting to the public hostname even though the Service has ready endpoints in the cluster, indicating routing reaches ingress but not the pod.
• •Health monitoring reports the deployment as Available but the application has not logged a single request after a rolling update, suggesting readiness remains false throughout.
• •Logs show connection refused or reset between proxy and upstream while the container is reporting healthy inside the node, pointing at port mismatch between Service and container.
• •Production traffic suddenly returns 502 after a configuration change to ingress, Service, or pod template, even though no image or replica count was altered.

Likely causes

• •Service targetPort does not match the containerPort the application actually binds to, so the kube-proxy forwards traffic to a port the container is not listening on.
• •Readiness probe returns non-2xx so endpoints are removed from the Service; the Deployment is Available per replica count but the Service has zero ready endpoints, producing upstream connect errors at the ingress.
• •Ingress backend service or port misroutes to a Service that exists but has no matching selector, so the upstream is empty and the controller returns 502.
• •Container crashed after start or is stuck in ImagePullBackOff or CrashLoopBackOff due to missing configuration, secrets, or wrong entrypoint; external surface still resolves but upstream is empty.
• •Network policy, security group, or pod-to-pod egress rule blocks ingress controller traffic from reaching pod IP on the container port, causing connection reset at the proxy boundary.
• •External load balancer health check path returns non-2xx and the load balancer drains the backend, so traffic is dropped at the edge with no application response.

First ten minutes

1. 01Reproduce the failure from outside the cluster using the public hostname and record the exact HTTP status line and any Server or Via header from the edge, so the 502 origin (edge vs origin) is identified before any internal change.
2. 02Confirm the Service exists for the expected selector and label set, then verify it has ready endpoints; zero ready endpoints immediately explains a 502 even when pods appear Running.
3. 03Compare the Service targetPort against the containerPort declared by the pod template and the listening port observed inside the container, so a port alias is caught before any restart cycle.
4. 04Read the most recent readiness probe outcome for each pod, including last error and last transition timestamp, to determine whether readiness is gating traffic or the application simply fails to serve.
5. 05Inspect ingress or load balancer backend configuration to confirm the upstream service name and port match the actual Service object, ruling out a routing misconfiguration independent of the Deployment.
6. 06Capture one network capture or connection log between proxy and pod IP on the configured port to determine whether the connection is refused, reset, or simply not attempted.

Evidence to collect

• •Exact HTTP status line, response headers including Server and Via, and body excerpt from the public probe, to distinguish a 502 produced at the edge from one produced at the application origin (reference MDN 502).
• •Service descriptor including selector labels, ports block, and session affinity, together with the corresponding endpoints or endpoint slices list showing ready addresses and ports.
• •Pod manifest fields containerPort, readinessProbe, livenessProbe, and startupProbe definitions alongside the most recent probe results and container state transitions.
• •Ingress or Gateway backend configuration with the routed service name, port number, and any path or host filters that could exclude the probe path.
• •External load balancer health check definition including protocol, port, path, and threshold values, and the corresponding backend health state.
• •Network policy, security group, or CNI restriction list relevant to ingress-to-pod traffic on the service port, plus any recorded connection refused or reset events from the proxy logs.

Where to look

• •The edge boundary: ingress controller logs and load balancer access logs where upstream connect errors and empty upstream responses are surfaced as 502 events.
• •The service boundary: Service and Endpoints or EndpointSlice objects, where a non-empty endpoint list with matching labels and ports is the contract for traffic delivery.
• •The pod boundary: container status, last termination reason, readiness and liveness probe history, and the bound ports reported by the container runtime inside the pod.
• •The configuration boundary: ingress backend and load balancer health check definitions, where misrouted service names or wrong health paths produce drained backends and 502s.
• •The network boundary: CNI policy, security groups, and node-level firewall rules between the ingress controller and pod IPs on the service port.
• •The build boundary: image registry and pull status, where a missing image, wrong tag, or registry credential failure leaves pods unschedulable or stuck in ImagePullBackOff despite a successful pipeline.

Diagnostic steps

1. 01Issue an HTTP request to the public hostname using a verbose client and capture status line, headers, and timing; a 502 from a CDN or load balancer with no upstream header indicates an edge or health-check drain condition, not an application bug (reference MDN 502).
2. 02List the Service and its endpoints, then compare selector labels to pod labels; if the selector matches but endpoints are empty, the most likely gate is readiness, not the Service object itself.
3. 03Diff the Service targetPort against the pod containerPort and the listening port inside the container; a port alias error produces a connection refused at the proxy while the pod appears Running.
4. 04Read probe status for each pod: if readinessProbe is failing, fix the probe path, port, or initial delay rather than restarting pods, because the same readiness failure will recur on the next rollout.
5. 05If endpoints are populated and ports match, validate ingress or gateway backend configuration: a route pointing to a service name with the right label but the wrong port number will produce an immediate 502 from the controller.
6. 06Check container state and recent termination logs: CrashLoopBackOff or Error status with non-zero exit codes after start indicates an application-side failure that no amount of routing change will fix.
7. 07Confirm the external load balancer health check path returns 2xx from within the cluster; if the path is correct internally but the edge shows 502, the load balancer is draining the backend and only configuration alignment will restore it.
8. 08If all of the above match, capture proxy-to-pod traffic on the service port to distinguish a refused connection (port wrong or pod down) from a reset (policy or network rule) before changing policy.

Common mistakes

• •Editing the Deployment to add replicas when the real failure is an empty endpoints list, which leaves the new replicas in the same blocked state and increases noise without changing outcomes.
• •Disabling the readiness probe to force endpoints to populate, which removes the only signal that distinguishes a booting application from a serving one and typically produces a transient 502 storm during rollout.
• •Restarting the ingress controller when the upstream service name in the backend is misspelled or points to a different namespace, since restarts do not re-read misrouted backends and the same 502 reappears on first request (reference MDN 502).
• •Bumping the image tag because the pipeline succeeded, when the actual image is fine but the readinessProbe path or port was changed by an unrelated edit and never corrected.
• •Adding allow-all network policies to bypass a connection reset, which masks the boundary at fault and can re-expose an unrelated service whose policy was the reason traffic was constrained.
• •Assuming a healthy external health check means the application is reachable, when the health check path differs from the user-facing path and the user-facing path is what produces the 502.

Safe fixes

• •If endpoints are empty despite running pods, correct the readinessProbe definition (path, port, initialDelaySeconds, periodSeconds, successThreshold) so the probe can return 2xx and endpoints are admitted, then re-check endpoints list rather than restarting the Deployment.
• •If Service targetPort does not match containerPort, update the Service port mapping or the pod containerPort declaration so they align, then confirm the application binds to the matching port inside the container before scaling any replicas.
• •If the ingress or gateway backend references a service name or port that does not match the actual Service, edit the routing object to point at the correct service and port, then re-issue an external HTTP probe and confirm 2xx before closing the task.
• •If the container is in CrashLoopBackOff, fix the application-level cause (missing environment, wrong entrypoint, failing migration) using the recorded termination logs, and only re-roll once the same container starts and serves on the declared port in a controlled pod.
• •If a network policy blocks ingress-to-pod traffic, narrow the policy to allow only the controller identity and the service port, then verify with a scoped test request rather than a blanket allow rule.
• •If the external load balancer is draining the backend because its health check path is wrong, point the health check at a path that returns 2xx from the same application code path as user traffic, then wait for the documented threshold before re-checking reachability.

Prove the fix

1. 01External HTTP probe to the public hostname returns a 2xx status with an expected payload, and the Server or Via header chain shows the request reached the application origin, not just an edge cache (reference MDN 502).
2. 02The Service reports a non-empty endpoints or endpoint slices list with the current pod IPs and the configured service port, and that list remains non-empty throughout a rolling update.
3. 03Container readiness and liveness probe history shows consistent success transitions during a fresh rollout, with no readiness flapping that would re-empty the endpoints list.
4. 04Ingress or load balancer access log shows successful upstream connections with 2xx response codes, and no 502 entries for the same client request patterns that previously failed.
5. 05An end-to-end synthetic check that exercises the user-facing path (not only the health check path) returns 2xx and the expected body, and remains green across at least one full restart cycle.
6. 06A rollback rehearsal confirms that reverting the Service, ingress, or probe change reproduces the original 502, demonstrating that the fix is causally tied to the observed symptom rather than incidental timing.

Prevention and next steps

• •Treat Deployment reachability as a contract: every change to image, containerPort, readinessProbe, Service ports, or ingress backend should be paired with an automated external probe check that fails the rollout if 2xx is not observed end to end.
• •Standardize probe path, port, and timeouts across services so a well-known health endpoint exists for load balancers and for the application itself, and review those values whenever the user-facing path changes.
• •Keep selector labels, Service ports, and ingress backend definitions under version control with the application manifest, and surface diffs of those files explicitly in the pull request so routing changes cannot drift from the deployment.
• •Capture edge, ingress, and container logs in a shared searchable store with request identifiers, so a 502 in production can be correlated to a specific upstream connection error in one investigation rather than three.
• •Run periodic chaos reachability drills that target one boundary at a time (wrong port, empty endpoints, blocked network policy) to verify the on-call playbook still maps symptom to boundary and that proof-of-fix steps remain valid.

Safe commands and checks

kubectl get svc -n <namespace> -l app=<app-label> -o yaml | sed -n '/^spec:/,/^status:/p'
kubectl get endpoints -n <namespace> <service-name> -o yaml
kubectl get pods -n <namespace> -l app=<app-label> -o jsonpath='{.items[*].status.containerStatuses[*].state}{.items[*].status.conditions[?(@.type==\"Ready\")].status}'
kubectl describe pod -n <namespace> <pod-name> | sed -n '/Containers:/,/Conditions:/p'
kubectl get ingress -n <namespace> -o jsonpath='{.items[*].spec.rules[*].http.paths[*].backend.service.name}{.items[*].spec.rules[*].http.paths[*].backend.service.port.number}'
kubectl logs -n <namespace> <pod-name> --previous --tail=200
kubectl get networkpolicy -n <namespace> -o yaml
kubectl rollout status deployment/<deployment-name> -n <namespace>