Kubernetes service-routing checklist

The symptoms

• •curl from inside the cluster to a Service ClusterIP returns connection refused, no route to host, or a non-2xx response from an unrelated pod, with the kubelet or kube-proxy showing no corresponding connection attempt for the intended pod.
• •The Service exists in the API with correct port and targetPort, but `kubectl get endpoints <svc>` shows 0 endpoints or a subset that does not match the Pod labels you expect.
• •`nslookup` or `dig` for the Service name resolves, but the returned A/AAAA records point to a ClusterIP that is not in the Service's `spec.clusterIPs`, or DNS returns NXDOMAIN inside the namespace while other Services resolve normally.
• •Traffic only fails for subsets of pods: ready pods with matching labels do not appear under Endpoints, or appear briefly and disappear after a relabel, indicating a label-selector mismatch rather than a network outage.
• •Same Service works from Node A but fails from Node B, suggesting a kube-proxy or CNI data-plane boundary on specific nodes rather than a control-plane routing policy.

Likely causes

• •Label-selector drift: Pod template labels changed via a rolling update, so `spec.selector` no longer matches `metadata.labels` on the live pods, and the Endpoints controller therefore stops reconciling the address set.
• •Readiness gate mismatch: Pods are Running and passing their own readinessProbe, but the Service targets a container `readinessProbe` that fails, so `endpoints.controller` removes them from the ready set, and kube-proxy programs iptables/IPVS without them.
• •EndpointSlice controller desync: the Endpoints object still appears populated while EndpointSlices are empty (or vice versa), often after a recent API server upgrade or when dual-stack family mismatches hide IPv4 or IPv6 targets.
• •kube-proxy data-plane lag: control plane looks correct, but iptables, IPVS, or nftables rules on a node have not been refreshed, so the ClusterIP forwards to stale endpoints until the dataplane reconverges.
• •DNS contract break: kube-dns/CoreDNS pods themselves are not Ready, or the Pod's `dnsPolicy` and `nameservers` search path has changed, so name resolution fails before the Service routing layer is even consulted.

First ten minutes

1. 01Confirm scope with evidence, not lore: pick one misrouted request path and capture the source pod IP, destination ClusterIP, destination port, and observed response so subsequent checks share a single failure boundary.
2. 02Verify the Service object contract: inspect `spec.selector`, `spec.ports[*].targetPort`, `spec.ipFamilies`, and `spec.internalTrafficPolicy`, then compare `spec.selector` to live pod labels with a label-equality query, not a substring match.
3. 03Read the Endpoints and EndpointSlices as the source of truth: the ready addresses must equal the set of pods that pass `spec.selector` AND have at least one ready container; any divergence is the candidate root cause.
4. 04Check kube-proxy health on the affected node(s): confirm the proxy has converged by inspecting its sync count, last sync timestamp, and the watch on the Service, before assuming the data plane is correct.
5. 05Validate DNS resolution from the source pod namespace: resolve the Service FQDN, confirm the returned IP is in `spec.clusterIPs`, and verify the search path if the Pod's `dnsPolicy` is not `ClusterFirst`.
6. 06Read the CNI boundary explicitly: confirm the ClusterIP falls inside the configured service CIDR and that node routes for that CIDR exist, so the routing failure is correctly attributed to the Service layer, not the underlay.

Evidence to collect

• •The full Service YAML with annotations, including `spec.selector`, `spec.ports`, `spec.ipFamilies`, `spec.internalTrafficPolicy`, and any `service.kubernetes.io/*` annotations that change routing behavior.
• •The current Endpoints object and the EndpointSlices that back it, with the `conditions.ready` and `conditions.serving` fields and the timestamp at which addresses were last added or removed.
• •Pod labels grouped by generation: the deployment's pod template labels versus the labels currently set on each running pod, so label drift across rollouts is visible in evidence rather than inferred.
• •kube-proxy sync state on each affected node: number of Services watched, last successful sync time, and any `iptables-restore` or `ipvs` write errors from the proxy log.
• •DNS resolution records from inside the source pod: A and AAAA records for the short name and the FQDN, the response status (NOERROR vs NXDOMAIN), and the TTL, paired with the pod's `dnsConfig` and `dnsPolicy`.
• •CNI evidence: the configured `--service-cluster-ip-range`, node route table entries for the Service CIDR, and the CNI plugin's logged errors for the affected node and namespace.

Where to look

• •API server boundary: `kubectl describe svc` and the raw Service object, because selectors, ports, IP families, and traffic policy are all encoded there and cannot be inferred from `get svc -o wide`.
• •Endpoints/EndpointSlices boundary: the `endpoints` resource, the `discovery.k8s.io` EndpointSlices, and the `endpoint-slice-mirror-xxx-*` controllers visible through `kubectl get events` for the Service.
• •Workload boundary: the `kubectl describe pod` output, specifically the `Ready` condition for each container, the labeled pod template of the owning controller, and any `Pod Readiness Gates` that change how Services observe readiness.
• •Data plane boundary: the kube-proxy log on the source and destination nodes, and the iptables counters, IPVS virtual server table, or nftables set entries that are produced by the proxy for the affected ClusterIP.
• •DNS boundary: the kube-dns or CoreDNS pods, their Service and Endpoints, and the `ndots`, `search` path, and `nameservers` actually mounted into the source pod's resolver configuration.

Diagnostic steps

1. 01Reconcile selector versus pod labels using a strict equality query, not a contains check; if any pod in the target set fails the equality test, that pod is not eligible to receive Service traffic regardless of Readiness.
2. 02Differentiate Endpoints emptiness from readiness: an Endpoints object with zero subsets means selector mismatch; an Endpoints object with subsets whose Pods are listed but readiness=false means the readinessProbe or readiness gate is the binding constraint.
3. 03Compare legacy Endpoints and EndpointSlices for divergence; if they disagree, escalate to whether the EndpointSlice controller is running and whether the Service has been re-created without a corresponding cleanup of stale slices.
4. 04On the source node, list iptables rules for the target ClusterIP (or `ipvsadm -ln` on IPVS clusters) and compare the destination set to the Endpoints list; a stale or empty rule set indicates kube-proxy convergence lag, not a Service definition error.
5. 05Resolve the Service from the source pod using the resolver mounted inside the container and confirm the returned IP is a member of `spec.clusterIPs`; an answer outside that set points to a DNS or CNI boundary rather than the Service object.
6. 06For each failed request, capture the SourceIP/DestIP/Protocol tuple from the workload's own access log and the receiving pod's listening socket, to prove whether the packet reached the cluster at all and whether the data plane is even programming the ClusterIP.

Common mistakes

• •Rewriting the Service selector to a broader label "to fix it" without first proving the label-drift hypothesis with evidence; widening the selector silently forwards traffic to pods that were never intended to receive it.
• •Restarting kube-proxy or CoreDNS pods on a single node and treating resolution as the proof of fix, when the underlying issue is a selector mismatch that the proxy restart cannot influence.
• •Trusting `kubectl get svc` output for routing correctness, because it reports the desired state of the Service but says nothing about Endpoints, kube-proxy convergence, or DNS reachability.
• •Adding a NodePort, LoadBalancer, or Ingress as a workaround before the ClusterIP path is proven, which obscures the original boundary and complicates rollback when the real Service definition is corrected.
• •Patching pod `readiness` to `true` manually to make traffic flow, because this skips the readiness gate that exists for a reason and will not survive a pod restart or controller resync.

Safe fixes

• •Correct a confirmed label-selector mismatch by editing the workload's pod template so the live pod labels equal the Service selector, then roll the deployment so the new pods are observed by the Endpoints controller; only do this if the live labels differ from the template and you can show the diff.
• •Restore readiness propagation by fixing the readinessProbe itself (endpoint, scheme, timeoutSeconds, periodSeconds) so the failing probe no longer removes pods from the Endpoints' ready set; do not bypass readiness without an explicit, documented exception.
• •Reconcile Endpoints and EndpointSlices by recreating the Service in a controlled manner when the controllers have desynced; only do this after backing up the current definition and confirming no client traffic depends on the existing Service UID.
• •Force kube-proxy data-plane convergence on a single node by restarting the kube-proxy after confirming the Endpoints object is correct; confine the restart to the affected node and verify the watch is re-established before declaring success.
• •Repair the DNS contract by fixing the Pod's `dnsPolicy` and `dnsConfig`, or by restoring the kube-dns/CoreDNS Service Endpoints; do not edit CoreDNS' Corefile as a substitute for addressing an unhealthy control-plane component.

Prove the fix

1. 01Repeat the original misrouted request from the same source pod and confirm the response is a 2xx (or the documented application success code) and the destination pod's access log shows the corresponding SourceIP, proving data plane traversal.
2. 02Show that `kubectl get endpoints <svc> --output yaml` lists all intended pods under `subsets[*].addresses`, with each address paired to a Ready pod, and that the count matches the Ready replica count of the owning workload.
3. 03On the source node, show the kube-proxy rule for the ClusterIP resolves to the intended pod IPs (or, on IPVS, that the virtual server's destinations are the corrected addresses) and that the rule was observed after the fix, not cached from before.
4. 04Resolve the Service FQDN from inside the source pod and confirm the returned A/AAAA record is in `spec.clusterIPs`, so DNS is no longer a confounding variable in the next incident.
5. 05Hold the verification through a rolling restart of the workload: with the same Service definition, prove that newly scheduled pods appear in Endpoints within the controller's expected window, demonstrating the fix survives the normal lifecycle rather than a one-shot correction.

Prevention and next steps

• •Treat `spec.selector` of every long-lived Service as an immutable contract: mandate that workload pod template labels are verified against it before any rollout, so the Endpoints controller cannot silently desynchronize from the Service definition.
• •Alert on Endpoints drift by monitoring the ratio of ready addresses to Ready replicas for each named Service; a sustained ratio below 1.0 for a StatefulSet/DaemonSet is a leading indicator before users notice the routing failure.
• •Pin kube-proxy version, CNI plugin, and CoreDNS version together in a tested matrix; mismatched data-plane versions are the most common cause of select-node-only failures that look like Service bugs but originate at the boundary.
• •Document pod readiness gates and any custom readiness probes in the Service's owning ticket so the next operator can distinguish readinessProbe failures from selector mismatches without re-deriving the intent each time.
• •Exercise DNS resolution and ClusterIP connectivity from a synthetic probe in each namespace at deploy time, so a routing regression surfaces at CI/CD rather than when a customer request misses its intended pod.

Safe commands and checks

kubectl get svc <svc> -o yaml -n <ns>
kubectl get endpoints <svc> -n <ns> -o yaml
kubectl get endpointslices -l kubernetes.io/service-name=<svc> -n <ns> -o yaml
kubectl get pods -l <key>=<value> -n <ns> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels}{"\n"}{end}'
kubectl -n <ns> exec <src-pod> -- nslookup <svc>.<ns>.svc.cluster.local
kubectl -n kube-system logs <kube-proxy-pod> --since=10m
kubectl -n kube-system logs <coredns-pod> --since=10m
kubectl get events -n <ns> --field-selector involvedObject.name=<svc>