Kubernetes readiness probe failed: find why traffic was withheld

The symptoms

• •kubectl get pods shows the pod in Running state with READY column showing 0/1 or 0/N, and the pod's Ready condition stays False.
• •kubectl describe pod reports Readiness probe failed or Readiness probe errored repeatedly in the Events list with HTTP status, connection refused, or timeout messages.
• •Application requests to the Service return 503, connection refused, or hang, while direct pod IP requests succeed when the probe target responds correctly.
• •kubectl get endpoints shows the pod IP missing from the Service's subset list, even though the pod is Running and its container has no restart count increase.
• •Rolling updates or scale-out events appear stuck because new pods never become Ready and the workload's readyReplicas stays below replicas.

Likely causes

• •The readiness probe path or port does not match the actual application listener, so the probe request gets connection refused or HTTP 404.
• •The probe initialDelaySeconds is shorter than the application's real warm-up time, causing the container to fail probes before it finishes initializing.
• •The probe periodSeconds or timeoutSeconds are too aggressive for a slow endpoint, producing intermittent failures under normal load.
• •Dependencies the readiness endpoint checks (database, cache, downstream API) are unavailable or slow, so the endpoint reports unhealthy even though the application process itself is fine.
• •A mismatched scheme (HTTP vs HTTPS) or a TLS handshake error between kubelet and the container causes every probe to fail before the application logic is even evaluated.
• •The pod is Ready=False because of a Pod-level readiness gate (such as a Pod Readiness Gate added by a service mesh or custom controller) rather than the container probe itself.

First ten minutes

1. 01Confirm the failure mode is readiness, not liveness or startup: run kubectl describe pod <pod> and read the Conditions block; Ready should be False while ContainersReady or Initialized may also be reported.
2. 02Capture the exact probe failure message from Events: grep the kubectl describe pod output for "Readiness probe failed" and note the HTTP status code, latency, or connection error string reported.
3. 03Verify the pod is excluded from Service traffic: run kubectl get endpoints <service> -n <namespace> and compare its addresses list to the pod IPs returned by kubectl get pods -l <selector>.
4. 04Read the last 100 lines of the container logs with kubectl logs <pod> --tail=100 to see whether the application started its listener before probes began; cross-check the probe's start time against log timestamps.
5. 05Reproduce the probe locally from the cluster network by using kubectl port-forward <pod> <local_port>:<probe_port> and issuing the same HTTP request the kubelet would send, to isolate whether the application or the probe definition is wrong.

Evidence to collect

• •Full output of kubectl describe pod <pod> -n <namespace>, including the Conditions array and the recent Events list.
• •The readyReplicas field of the owning Deployment, StatefulSet, or ReplicaSet compared to desired replicas, to quantify how many pods are stuck.
• •The addresses list of the target Service endpoints and the notReadyAddresses list, to confirm whether traffic is actually being withheld.
• •Container logs from the moment of first probe attempt through the most recent failure, including any startup banner that proves the listener bound to the expected port.
• •The Pod's effective readinessProbe, livenessProbe, and startupProbe definitions, including scheme, path, port, initialDelaySeconds, periodSeconds, timeoutSeconds, successThreshold, and failureThreshold.
• •If a Pod Readiness Gate is configured, the status of the corresponding condition (for example, service.beta.kubernetes.io/...) from kubectl get pod <pod> -o jsonpath='{.status.conditions}'.

Where to look

• •The kubelet-probe boundary: kubelet executes readiness probes against the pod IP and the containerPort, not against the Service virtual IP, so probe behavior is independent of kube-proxy rules.
• •The Pod spec boundary in the API server: readinessProbe is read from spec.containers[*].readinessProbe, so merged defaults from a Pod template and any higher-level controller must be inspected together.
• •The EndpointSlice or Endpoints object boundary: kube-controller-manager builds subsets from pods whose Ready condition is True, so any mismatch between pod state and Service subset points to this control loop.
• •The application listener boundary inside the container: probes target a specific port and path; verify the process binds to the expected interface and responds on the expected route.
• •The dependency boundary: if the readiness endpoint performs a downstream check, trace which external system it queries and confirm that dependency's health from inside the cluster network.

Diagnostic steps

1. 01Run kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.conditions}' and confirm the Ready condition status, reason, and message; Ready=False with reason ContainersNotReady indicates container probes failed.
2. 02Run kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].readinessProbe}' to capture the effective probe definition and compare each field (httpGet.path, httpGet.port, tcpSocket.port, exec.command, initialDelaySeconds, periodSeconds, timeoutSeconds, failureThreshold).
3. 03Run kubectl describe pod <pod> -n <namespace> and inspect the Events list for entries of type Warning with reason Unhealthy and message starting "Readiness probe failed:"; record the error string verbatim.
4. 04Run kubectl get endpoints <service> -n <namespace> -o yaml and compare subsets[*].addresses to the pod IP; absence of the pod's IP confirms traffic is withheld because Ready is False.
5. 05Run kubectl exec <pod> -- ss -ltnp (or netstat -ltnp) to confirm the application process is listening on the exact port the probe targets and on the interface expected by the pod network.
6. 06Run kubectl exec <pod> -- wget -qO- http://<pod-ip>:<probe_port><probe_path> (or curl) to reproduce the probe request inside the container and capture the response code and latency.
7. 07If exec probes are configured, run kubectl exec <pod> -- <exec_command> to reproduce the readiness command directly and inspect stdout, stderr, and exit code, then map failureThreshold successes or failures to the readiness state.
8. 08If a Pod Readiness Gate is present, run kubectl get pod <pod> -o jsonpath='{.status.conditions}' and inspect any custom condition whose type is not Ready, ContainersReady, Initialized, or PodScheduled; its status explains why Ready is False.

Common mistakes

• •Assuming a Running pod is serving traffic: Running only means the container process is alive; Ready=False removes the pod from Service subsets regardless of Running state.
• •Restarting the pod without reading the probe Events: restarts driven by liveness probes do not fix readiness failures, and kubelet will continue issuing the same failing probe against the new container.
• •Pointing the probe at the wrong port or path: a 404 from the application counts as a failure, but the Events message may not make the wrong path obvious; always cross-check the spec against the actual listener.
• •Setting initialDelaySeconds too low: a warm-up period of several seconds for JVM, Python, or Node.js startup will cause every probe to fail until the listener is ready.
• •Embedding expensive downstream checks in the readiness endpoint: if the endpoint queries a database or third-party API, that dependency's blip will pull every pod out of rotation at once.
• •Confusing startupProbe failures with readinessProbe failures: during the startup window kubelet suspends readiness and liveness checks, so a failing readiness probe only after the startup window proves is a readiness issue, not a startup issue.

Safe fixes

• •If the probe path or port is wrong, correct spec.containers[*].readinessProbe.httpGet.path or httpGet.port in the Pod template (often via the workload's Pod spec), redeploy, and confirm a new pod reports Ready=True within periodSeconds.
• •If initialDelaySeconds is shorter than observed startup time, increase initialDelaySeconds to at least the time between container start and first successful listener log line; redeploy and watch for Ready=True.
• •If timeoutSeconds is shorter than observed probe latency under normal load, raise timeoutSeconds above p99 probe latency measured from the Events timestamps.
• •If the readiness endpoint checks a flaky downstream dependency, narrow the probe to a lightweight local check (such as a /healthz that returns 200 once the listener is up) and move deep dependency checks to a separate diagnostic path.
• •If scheme mismatch or TLS handshake causes failures, align readinessProbe.httpGet.scheme with the actual listener (HTTP vs HTTPS) and confirm any required client certificates are mounted and trusted.
• •If a Pod Readiness Gate is keeping Ready=False, address the upstream controller that owns the gate (for example, a service mesh sidecar readiness signal) rather than editing kubelet probe fields.

Prove the fix

1. 01After the change rolls out, run kubectl get pods -l <selector> -n <namespace> and confirm the READY column shows 1/1 for every replica, and readyReplicas on the owning workload equals replicas.
2. 02Run kubectl describe pod <pod> -n <namespace> and confirm the Ready condition status is True with reason "", and that no new Warning events of reason Unhealthy mention Readiness probe failed since the redeploy.
3. 03Run kubectl get endpoints <service> -n <namespace> -o yaml and confirm subsets[*].addresses contains the IP of every replica and subsets[*].notReadyAddresses is empty.
4. 04Send a synthetic request to the Service DNS name on the Service port (for example via a debug pod using getent and a request tool) and observe a 2xx response from any replica, repeated across several minutes to confirm steady membership.
5. 05Trigger a rolling restart of the workload and confirm each new pod reaches Ready=True within the configured initialDelaySeconds plus a small buffer, and that readyReplicas tracks replicas throughout the rollout.

Prevention and next steps

• •Define readiness probes that probe only the local listener, not external dependencies, and keep separate liveness and diagnostic endpoints to avoid coupling rotation to downstream health.
• •Use a startupProbe when cold start times are variable, and set its failureThreshold generously so readiness and liveness probes only activate after the application is truly ready.
• •Capture baseline probe latency in non-production environments and set timeoutSeconds and periodSeconds above observed p99, so normal request variance does not produce false failures.
• •Alert on the difference between desired replicas and readyReplicas, and on the count of pods with Ready=False for more than a configurable threshold, so readiness withholding is detected before users are affected.

Safe commands and checks

kubectl describe pod <pod> -n <namespace>
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.conditions}'
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].readinessProbe}'
kubectl get endpoints <service> -n <namespace> -o yaml
kubectl get pods -l <selector> -n <namespace> -o wide
kubectl logs <pod> -n <namespace> --tail=100 --timestamps=true
kubectl exec <pod> -n <namespace> -- ss -ltnp
kubectl exec <pod> -n <namespace> -- wget -qO- http://<pod-ip>:<probe_port><probe_path>