Kubernetes startup probe race: diagnose checks that run before readiness

The symptoms

• •Container is restarted repeatedly during the first seconds of life while CPU and memory are low, and `Last State: Terminated` shows `Reason: Completed` or `Exit Code: 0/1` before the first successful readiness check.
• •Readiness probe never reports Ready even though the process is running and bound to the port, and endpoint controller keeps the Pod out of Service endpoints until the startup window passes.
• •Application logs show the listening socket open, but downstream dependencies (database pool, config map parser, feature flag fetcher) finish initialization seconds after the process accepts traffic.
• •Restart count resets only after the probe contract changes; with a `startupProbe` present, the kill happens before `startupProbe.successThreshold` is satisfied, not during steady-state operation.
• •Metrics show probe latency for the first probe attempts far higher than later ones, indicating the check is timing out against an uninitialized handler rather than a slow steady-state handler.

Likely causes

• •The application's "ready" contract is wired to the HTTP listener rather than to dependency readiness, so the probe returns 200 while migrations, cache priming, or auth token exchange are still in progress.
• •A `startupProbe` is absent or too short, so `livenessProbe` or `readinessProbe` begins evaluating against a process that has not yet finished its initialization contract.
• •Sidecar ordering (Istio, OpenTelemetry, secrets-init) starts the application container before mounted config, certificates, or service account tokens are usable, so the probe checks a half-configured process.
• •PreStop hooks, init containers, or postStart handlers run after the main container's listener is up, decoupled from the probe timeline, so the readiness surface is evaluated before initialization completes.
• •The probe handler itself is an unauthenticated `/healthz` that short-circuits on process state but does not validate the same dependencies the data path requires, exposing a gap between probe semantics and runtime contract.

First ten minutes

1. 01Confirm the Pod is in `CrashLoopBackOff` or has a high `restartCount` only during the first window after creation, using `kubectl describe pod <pod>`. Note the `Last State` and `Started` timestamps.
2. 02Read the container's startup logs in order (init container logs, then main container logs) to identify the first event marked "ready", "listening", or "initialized"; compare that timestamp to the probe's first evaluation.
3. 03List the probe definitions on the Pod spec and record `initialDelaySeconds`, `periodSeconds`, `failureThreshold`, `timeoutSeconds`, and `successThreshold` for each of startup, readiness, and liveness.
4. 04Watch `kubectl get pod <pod> -w` and capture the transitions for `Ready` and `ContainersReady`; a Ready=False that never flips before a restart is the canonical signature of a startup probe race.
5. 05Inspect the readiness endpoint manually with a one-shot exec probe from inside the Pod to confirm whether the handler returns 200 before the application finishes its dependency handshake.

Evidence to collect

• •Full Pod describe output including `Conditions`, `Container Statuses`, `Events`, and `Last State` reason codes, with monotonic timestamps.
• •Ordered application logs from the first process start, with logger timestamps aligned to the probe timeline; record the first "ready" or "serving" line.
• •Probe definitions as rendered by `kubectl get pod <pod> -o jsonpath` for the readiness, liveness, and startup probes, including all timing fields.
• •Init container logs and sidecar readiness states, since a sidecar that fails readiness can hold the Pod out of Service endpoints and surface as a startup race in user traffic.
• •EndpointSlice or Endpoints object state for the owning Service, captured with `kubectl get endpointslices -l kubernetes.io/service-name=<svc>` to confirm whether the Pod ever appeared as a backing target.

Where to look

• •Pod boundary: `Conditions`, `ContainerStatuses[*].State`, `LastState`, and `Events` produced by the kubelet on the node hosting the Pod.
• •Container boundary: stdout/stderr from the main container, init containers, and any inject sidecar (Istio, Vault, secrets-init), ordered by logger timestamp.
• •Service boundary: the EndpointSlice controller's view of which Pods are Ready, and the time of the first transition into the Service's endpoint set.
• •Probe boundary: the kubelet's probe recorder, which writes `Liveness`, `Readiness`, and `Startup` events on the Pod and exposes timing per attempt.
• •Application boundary: the code path that defines the readiness handler, distinguishing "process is up" from "dependencies are usable".

Diagnostic steps

1. 01Annotate the timeline: Pod creation time, first probe evaluation time, first "ready" log line, and first `Ready: True` transition. If the first probe runs before the first "ready" log line, a startup race is the leading hypothesis.
2. 02Compare probe timing fields against the application's measured cold-start duration across at least three restarts; if `initialDelaySeconds + (failureThreshold × periodSeconds)` is shorter than the observed cold start, the contract is unsafe.
3. 03Reproduce the race in a controlled rollout with `kubectl rollout pause`/`resume` and a single replica, observing the probe event timestamps; this isolates whether the failure is cluster-wide or tied to a specific node.
4. 04Cross-check the readiness handler logic against the data path: any dependency the request handler queries (database, cache, downstream service) that the probe does not query is a candidate gap.
5. 05Verify whether a `startupProbe` exists; if it does, compute the maximum startup window it permits and compare to the application's longest cold-start dependency. If `startupProbe` is absent, treat the readiness probe as the boundary under suspicion.
6. 06Inspect init container ordering and postStart handlers; if a postStart runs asynchronously after the main PID is alive, the readiness surface can be evaluated before postStart completes.
7. 07Rule out node-level causes: CPU throttling, image pull latency, and seccomp or AppArmor denials can present similarly but originate outside the probe boundary.

Common mistakes

• •Adding a generous `initialDelaySeconds` to the readiness probe as a workaround, which masks the race but leaves liveness on the same uncapped window and can allow an actually-broken container to stay in service.
• •Pointing the probe at the same `/healthz` endpoint used for liveness without verifying that handler covers the same dependencies the request path requires, producing a false Ready signal.
• •Confusing a startup race with steady-state liveness flapping; the diagnostic is the first window only, not the long tail, so evidence must be scoped to the first 30-120 seconds of life.
• •Assuming the readiness endpoint returns 200 from inside the Pod implies it is reachable from the kubelet; the kubelet probes over the container's network namespace, which may differ from cluster network policy.
• •Increasing `failureThreshold` to swallow the race, which delays the kill but does not change the underlying contract and can hold a half-initialized Pod in Service endpoints.

Safe fixes

• •Introduce a `startupProbe` whose `failureThreshold × periodSeconds` covers the measured worst-case cold start, leaving `livenessProbe` and `readinessProbe` to handle steady-state semantics; this is the documented separation of concerns per the Kubernetes debugging application guide.
• •Tighten the readiness handler to gate on the same dependency contract the request path requires (database pool, config loaded, downstream handshake complete), so the probe signal mirrors what traffic would experience.
• •Move work that is prerequisite to readiness (cache warm-up, schema migrations, token fetch) into init containers or a pre-`signal` startup script, so the main container's PID is not reported Ready until those prerequisites complete.
• •Decouple the probe port from the data port only when the probe handler is intentionally narrower than the data path; document the contract so the divergence is not accidental.
• •Capture cold-start duration as a metric (probe histogram latency on the first N attempts) and alert when p95 of the first attempt exceeds the `startupProbe` window, giving early warning of regression.

Prove the fix

1. 01Deploy a fixed manifest with a `startupProbe` that covers the measured cold start, observe at least three full restart cycles, and confirm `restartCount` does not increment during the startup window.
2. 02Verify the readiness condition transitions to `True` only after the application's "ready" log line, by aligning log timestamps with the first `Ready: True` event time.
3. 03Confirm the EndpointSlice picks up the Pod within the expected window of the readiness probe going Ready, and that traffic shifts to the Pod without 5xx attributed to initialization.
4. 04Re-run a controlled failure injection (kill the dependency the probe gates on) and confirm the Pod is removed from Service endpoints before user traffic observes errors, proving the probe now reflects the real contract.
5. 05Revert the fix on a single replica and observe the race signature returns (early `Ready: False`, no eventual `Ready: True`, restart before the first success), establishing a reversible regression check.

Prevention and next steps

• •Define a written "ready contract" for each service that lists which dependencies must be initialized before the probe returns Ready, and gate the probe handler on that contract explicitly.
• •Standardize a `startupProbe` template per service family with `failureThreshold` sized to the worst-case cold start plus a safety margin, and reject manifests that omit it in admission policy.
• •Emit a structured "ready" log line with a monotonic timestamp and require it to appear before any readiness transition in dashboards, so regressions are visible at deploy time.
• •Track probe latency histograms split by attempt order; anomalies on the first attempts are the leading indicator of a developing startup race.
• •Review init container ordering and postStart handlers at PR time, since any work that runs after the main PID becomes reachable by the probe is a latent race.

Safe commands and checks

kubectl describe pod <pod> -n <namespace>
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].readinessProbe}{.spec.containers[*].livenessProbe}{.spec.containers[*].startupProbe}'
kubectl logs <pod> -n <namespace> --all-containers --timestamps --tail=200
kubectl get pod <pod> -n <namespace> -w -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.lastTransitionTime}{"\n"}{end}'
kubectl get endpointslices -l kubernetes.io/service-name=<service> -n <namespace> -o yaml
kubectl exec <pod> -n <namespace> -- /bin/sh -c 'command -v wget >/dev/null && wget -qO- http://<pod-ip>:<port>/healthz || command -v curl >/dev/null && curl -fsS http://<pod-ip>:<port>/healthz || echo no-http-client'
kubectl rollout pause deployment/<deployment> -n <namespace> && kubectl rollout resume deployment/<deployment> -n <namespace>