How to verify Kubernetes readiness removes and restores traffic correctly

The symptoms

• •Traffic continues to be sent to a Pod that shows a NotReady condition or has failing readiness events in its status, indicating the probe is not gating Endpoints.
• •After a transient backend failure the Pod never regains traffic, even though its container is healthy and responding, indicating stale NotReady state or a never-recovered probe.
• •Readiness probe events in the Pod lifecycle show repeated Failed, Unhealthy, or Back-off messages that do not match the application's actual response times.
• •kubectl describe endpoints shows Pod IPs that are not in the Pod's own Ready status, or the endpoints list does not change when probes flip.
• •Client-side error metrics (for example 5xx rate or connection refused) cluster on specific Pod IPs that Endpoints still advertise.

Likely causes

• •Probe points at the wrong port, path, or scheme, so it always fails or always succeeds regardless of application health.
• •initialDelaySeconds is too short for a slow JVM, cache warm-up, or large image pull, producing cascading false negatives during startup.
• •failureThreshold, timeoutSeconds, or periodSeconds are tuned so that a single slow request removes the Pod and the Pod never accumulates enough successes to recover.
• •The Service selector does not match the Pod labels, so Endpoints are empty or stale and traffic never reflects probe state.
• •NetworkPolicy, missing ServiceAccount, or a broken readinessExec command prevents the kubelet from evaluating the probe at all, leaving the Pod in a phantom Ready state.
• •readinessProbe and livenessProbe are confused: a liveness failure restarts the container while readiness silently fails, so Endpoints stay populated during the restarts.

First ten minutes

1. 01Confirm the symptom class: reproduction shows traffic arriving at a Pod whose Ready condition is False, or a Pod that never reappears in the Endpoints after recovery.
2. 02Identify the exact Service and Pod under test by name and namespace; record the Service selector labels and the Pod's labels so they can be compared.
3. 03Inspect the Pod's current readiness condition and the recent events stream to see whether the probe is the source of the NotReady state.
4. 04Inspect the Endpoints (or EndpointSlice) object for the Service and list the subset of Pod IPs currently advertised for the Service port.
5. 05Compare the Endpoints subset with the Pod's Ready condition; any divergence is the primary evidence that readiness is not gating traffic.
6. 06Capture one sample of the probe target (path, port, scheme) directly from the Pod spec so the test can be replayed safely out of band.

Evidence to collect

• •Pod Ready condition value, lastTransitionTime, and the reason/message reported by the kubelet.
• •Sequence of readiness-related events from the Pod, including type, reason, and message such as Unhealthy, Failed, or Back-off.
• •EndpointSlice subsets for the Service, listing each address and the conditions ready, serving, and terminating.
• •Pod spec fields httpGet, tcpSocket, exec, initialDelaySeconds, periodSeconds, timeoutSeconds, successThreshold, and failureThreshold.
• •Timestamp of the last successful readiness check, derived from kube-probe or container status fields, to bound the recovery window.
• •Client-side observation of which Pod IP serviced a given request, derived from access logs or sidecar telemetry, correlated with the Endpoint subset at that time.

Where to look

• •api-server boundary: the EndpointSlice and Endpoints objects are the source of truth that kube-proxy consumes; probe state must be reflected here within a few seconds.
• •kubelet-probe boundary: the kubelet runs readiness probes on the node and writes the result back as a Pod condition; the events stream surfaces each failure.
• •Pod-spec boundary: the readinessProbe block on the container, plus the corresponding Service selector, defines the contract that the runtime is supposed to enforce.
• •application boundary: the probe target (HTTP path, TCP port, or exec command) must be representative of the dependency the Service actually needs upstream.
• •network-policy boundary: egress restrictions on the node or namespace can prevent a TCP or HTTP probe from reaching the container, mimicking application failure.

Diagnostic steps

1. 01Describe the Pod and read the Ready condition; if it is False, note the reason, the lastTransitionTime, and whether the message mentions readiness, liveness, or startup.
2. 02Describe the Service and capture its selector; compare against the Pod's labels using a label query to confirm the Pod is in the Service's intended set.
3. 03Read the Endpoints and EndpointSlice for the Service and enumerate the addresses; cross-check each address against the Ready condition of the corresponding Pod.
4. 04Replay the readiness probe against the Pod's IP from outside the Pod (for example, an HTTP GET to the recorded path and port) and record latency and status; this validates the probe target itself.
5. 05Tail the events stream filtered to the Pod and look for repeated Unhealthy or Failed events with their timestamps; correlate the first event with the lastTransitionTime on the Ready condition.
6. 06Compare the probe configuration against the application's measured startup and steady-state latency; mark whether initialDelaySeconds, periodSeconds, and failureThreshold are consistent with observed behavior.
7. 07During a controlled failure, observe whether the Pod IP disappears from the EndpointSlice within one periodSeconds interval; if it remains, the probe is not gating traffic.
8. 08After recovery, observe whether the Pod IP reappears in the EndpointSlice within one periodSeconds + successThreshold × periodSeconds interval; if it does not, the Pod is permanently excluded.

Common mistakes

• •Trusting the Service alone and assuming readiness is enforced; the Endpoints object is the real authority and must be checked directly.
• •Confusing livenessProbe with readinessProbe: a liveness failure restarts the container and does not directly remove the Pod from the Endpoints list, so a confusing spec can leave a broken Pod in rotation.
• •Pointing readiness at a database or queue that is briefly unavailable; a single failure should not require many successes to recover, but a mis-set successThreshold can.
• •Setting periodSeconds shorter than the probe's worst-case latency, so even a healthy application accumulates failures and is removed from rotation.
• •Using a readiness probe that hits a cache rather than the dependency that the Service actually depends on, so the Pod reports Ready before it can serve real traffic.
• •Looking only at the Pod's phase and ignoring the Ready condition, which is the field that drives Endpoint membership.

Safe fixes

• •If the probe target is wrong, correct the httpGet path and port (or tcpSocket port) to match the actual health endpoint the application exposes, then re-run the verification sequence.
• •If initialDelaySeconds is too short for the application's startup, raise it to a value that exceeds the observed cold-start time, evidenced by the first successful readiness event.
• •If failureThreshold is too low for an upstream dependency that is briefly unavailable, raise it to a value that tolerates the observed upstream outage duration, then re-run the controlled failure.
• •If the Service selector does not match the Pod labels, align the selector with the Pod's actual labels and confirm the Endpoints list becomes non-empty within the next reconciliation cycle.
• •If a NetworkPolicy blocks the probe, adjust the policy to allow kubelet source ranges to reach the readiness port, then re-check the events stream for Unhealthy events.
• •If readiness and liveness are confused, separate them: readiness should reflect dependency availability, liveness should reflect deadlock or unrecoverable internal state, and both should be re-verified end-to-end.
• •Each fix must be followed by the proof-of-fix sequence below; do not declare success on spec change alone.
• •All changes must be applied through a controlled rollout (for example, a Deployment update with a small replica count) so that a regression is contained to the test population.

Prove the fix

1. 01Inject a controlled failure into the readiness target (for example, return a non-2xx status from the readiness endpoint) and verify, within periodSeconds × failureThreshold, that the Pod's IP is removed from the EndpointSlice subset and that no client request reaches the Pod IP during the failure window.
2. 02Restore the readiness target to a healthy state and verify, within periodSeconds × successThreshold, that the Pod's IP reappears in the EndpointSlice subset and that subsequent client requests reach the Pod IP again.
3. 03Confirm the Pod's Ready condition flips from False to True with a matching lastTransitionTime within the same recovery window, and that the events stream contains a corresponding successful probe event.
4. 04Repeat the failure and recovery cycle at least twice to confirm the behavior is not a one-off; reject the fix if any cycle leaves the Pod in a stale NotReady state or in a stale Ready state.
5. 05Run the same sequence against a second Pod in the same Service to confirm the behavior is not specific to one replica whose local state diverged from the cluster-wide view.
6. 06Record the pre-change and post-change Endpoints subsets and Ready conditions as evidence; if any post-change observation does not match the expected behavior, treat the fix as unproven and reopen the diagnosis.

Prevention and next steps

• •Treat readiness as a tested contract: every change to the readiness probe spec, the Service selector, or the readiness endpoint implementation must be followed by the controlled failure and recovery verification.
• •Keep readiness probe targets independent of upstream dependencies that are expected to be briefly unavailable; prefer a process-local endpoint that reflects true service readiness.
• •Tune initialDelaySeconds, periodSeconds, timeoutSeconds, failureThreshold, and successThreshold from observed latency and failure data, not from defaults; revisit the tuning after each change to dependencies or startup paths.
• •Add a synthetic check that exercises the readiness contract end-to-end on every cluster upgrade and on every Service template change, so regressions are caught before user traffic is affected.
• •Add an alert on Pods that have been NotReady for longer than a defined budget, since persistent NotReady pods indicate a probe, target, or selector problem that needs human investigation.
• •Document the expected Endpoints subset size and shape for each Service, so that a divergence between the Service selector's match and the EndpointSlice subset is detected early.

Safe commands and checks

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.conditions[?(@.type=="Ready")]}': print the Ready condition including status, reason, message, and lastTransitionTime for the Pod under test.
kubectl describe pod <pod-name> -n <namespace>: read the events stream and look for Unhealthy, Failed, or Back-off entries that mention readiness, liveness, or startup probes.
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].readinessProbe}': print the readinessProbe block (httpGet, tcpSocket, exec, initialDelaySeconds, periodSeconds, timeoutSeconds, failureThreshold, successThreshold) for verification against the application's measured behavior.
kubectl get svc <service-name> -n <namespace> -o jsonpath='{.spec.selector}': print the Service selector so it can be compared against the Pod's labels; this identifies selector mismatch.
kubectl get pod <pod-name> -n <namespace> --show-labels: print the Pod's labels so the Service selector can be matched against them directly.
kubectl get endpoints <service-name> -n <namespace> -o yaml: list the addresses and ports currently advertised for the Service; this is the source of truth that kube-proxy consumes.
kubectl get endpointslices -l kubernetes.io/service-name=<service-name> -n <namespace> -o yaml: list the EndpointSlice subsets for the Service, including per-address conditions ready, serving, and terminating, for higher-resolution verification.
kubectl get pods -n <namespace> -l <service-selector-key>=<service-selector-value> -o jsonpath='{range .items[*]}{.metadata.name}{" ready="}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}': enumerate the Pods that match the Service selector and their Ready status, so the Service's intended set can be compared with the Endpoints subset.