Kubernetes CrashLoopBackOff: distinguish process exit from probe failure

The symptoms

• •kubectl get pods shows a pod stuck in CrashLoopBackOff with a rising RESTART COUNT and a Last State of either Error or Terminated with a non-zero exit code.
• •kubectl describe pod reports Back-off restarting failed container with a Last State reason of Error, and the Events section shows Back-off pulling, Back-off restarting, or Failed with details about an exec probe, HTTP probe, or TCP probe.
• •Logs from the previous instance (kubectl logs --previous) terminate abruptly without a clean shutdown line, or stop at the moment a readiness or liveness probe began failing.
• •The pod reaches the Running phase momentarily before re-entering CrashLoopBackOff, with probe-related entries preceding each restart in the event timeline.
• •Multiple containers in the same pod restart independently with different Last State reasons, indicating the failing container can be isolated by name from -c <container>.

Likely causes

• •Process-level crash: the container's main process exited with a non-zero status during initialization, often visible as Error in Last State with an Exit Code distinct from the probe-related codes 137 or 143.
• •Probe misconfiguration: a livenessProbe or readinessProbe uses the wrong handler, the wrong port, or an initialDelaySeconds/periodSeconds/timeoutSeconds/failureThreshold combination that declares the process unhealthy before it has finished initializing.
• •Probe target unavailable: the probe points at an HTTP path, TCP port, or exec command that depends on a sidecar, mounted secret, config map, or service that is not yet present in the pod's network or filesystem namespace.
• •Resource starvation at boot: requests below the actual memory or CPU footprint produce an OOMKill during startup, surfaced as Exit Code 137 in Last State and as an OOMKilled true annotation rather than as a probe failure.
• •Image pull or entrypoint drift: an initContainer or container command references an image tag that resolved differently after a registry rollout, producing repeated process exits independent of probe behavior.

First ten minutes

1. 01Run kubectl get pod <pod> -n <namespace> -o wide and record RESTART COUNT, the Last State column, and the node name; treat any RESTART COUNT above zero as evidence the container lifecycle contract has already been broken once.
2. 02Run kubectl describe pod <pod> -n <namespace> and locate the failing container in the Containers section, then capture Last State, Reason, Exit Code, and Started At to determine whether the kubelet recorded a process exit or a probe failure as the restart trigger.
3. 03Scroll to the Events section of the same describe output and capture the ordered lines containing the substrings Back-off, Failed, Killing, Unhealthy, or ProbeFailed, since each one names the actor that triggered the most recent restart.
4. 04Run kubectl logs <pod> -n <namespace> --previous --tail=200 for the failing container and stop reading at the last line emitted before the container terminated, because anything after that line is post-mortem noise.
5. 05Cross-check Reason and Exit Code against the Kubernetes documentation on container lifecycle hooks and probes so that probe-induced restarts are not mistaken for application crashes and vice versa.

Evidence to collect

• •The Reason field under Containers[].state.terminated or Containers[].lastState.terminated for the failing container, paired with the numeric Exit Code recorded by the kubelet.
• •The ordered Events list from kubectl describe pod, filtered to entries whose reason contains Back-off, Failed, Killing, Unhealthy, or ProbeFailed and whose message references the container name.
• •The last 200 lines of kubectl logs --previous for the failing container, plus the timestamp of the final log line relative to the Started At field in Last State.
• •The OOMKilled, Reason, and Exit Code triple for each container when memory or CPU limits are a candidate cause, since Exit Code 137 combined with OOMKilled true points to cgroup pressure rather than a probe.
• •The exact Liveness, Readiness, and Startup probe definitions from the pod spec, including handler type, path or port, initialDelaySeconds, periodSeconds, timeoutSeconds, successThreshold, and failureThreshold.

Where to look

• •The Containers array inside kubectl describe pod <pod> -n <namespace>, specifically the state and lastState fields for the named container, which is the kubelet's authoritative record of what terminated the process.
• •The Events section of the same describe output, since the kubelet emits one event per restart attempt and per failed probe with a reason field that distinguishes probe failures from image or runtime failures.
• •The pod spec stored in etcd, accessible via kubectl get pod <pod> -n <namespace> -o yaml, where the probe definitions and their thresholds reveal whether the probe contract is realistic for the application's startup time.
• •The previous container's log stream, accessed through the kubelet's previous-instance log path or kubectl logs --previous, since the last lines before termination often encode the root cause that the restart obscures.
• •Node-level journal records when the container runtime itself rejected the start, because kubelet events may summarize the failure and the node log retains the underlying runtime error message.

Diagnostic steps

1. 01Compare Last State.Reason to the documented values Error, Completed, OOMKilled, and ContainerStatusUnknown; only Error with a non-zero exit code implicates the application's own process, while OOMKilled and Completed indicate kernel and lifecycle signals respectively.
2. 02Match Exit Code 137 to cgroup OOMKill evidence and Exit Code 143 to a SIGTERM delivered during graceful shutdown, so probe-induced kills are not misattributed to application defects.
3. 03Filter describe events for the substring ProbeFailed: a ProbeFailed event whose message names httpGet, tcpSocket, or exec identifies a misbehaving probe and rules out a process crash as the proximate cause.
4. 04Reconstruct the timeline by interleaving Events timestamps with kubectl logs --previous line timestamps, so the operator can see whether the process stopped on its own or was killed while still producing output.
5. 05Inspect the probe definition for each container and verify that the handler port or path exists in the container image at startup; a probe whose target is only created by a sidecar or by a later migration step will fire against an absent contract.
6. 06Decide between process exit and probe failure by checking whether the Last State was recorded before or after a ProbeFailed event; events ordered Killing followed by Back-off indicate the kubelet restarted on probe signal rather than on process exit.

Common mistakes

• •Reading only kubectl get pods and concluding that any CrashLoopBackOff pod has a crashing application, when the kubelet often enters that phase after repeated probe failures with no process-level crash at all.
• •Assuming kubectl logs --previous will explain every CrashLoopBackOff, since a probe-induced kill leaves no application-side stack trace and the absence of log lines near the kill time is itself evidence of a probe-driven restart.
• •Setting failureThreshold very high to silence probe failures, which masks the contract violation and lets an unhealthy container serve traffic until the kubelet finally acts.
• •Treating every non-zero Exit Code as a crash, when Exit Code 137 from OOMKilled and Exit Code 143 from SIGTERM are signals whose root cause lies outside the application's source code.
• •Adding initialDelaySeconds without measuring real startup time, which delays the probe but does not address a probe whose handler targets a port or path that the application never opens.

Safe fixes

• •If Last State.Reason is Error and Exit Code is non-zero, address the application crash by reading the last lines of kubectl logs --previous for the failing container and reproducing the boot sequence locally with the same environment variables and mounted config.
• •If events show ProbeFailed and the probe handler is exec, replace the exec command with one that exits zero at readiness and rerun the probe manually inside a debug copy of the pod spec to confirm the handler contract.
• •If the probe is httpGet on a port that is bound late in startup, raise initialDelaySeconds only after measuring actual readiness with a timed probe from inside a debug container in the same pod.
• •If OOMKilled is true, raise the memory limit and request to a value documented above the observed working-set size, and reapply the change as a rolling update so the prior pod is not deleted in place.
• •If a startupProbe is not present and startup time is variable, add a startupProbe whose failureThreshold tolerates slow boot and which gates the livenessProbe from running until startup succeeds, preventing premature restart.
• •If the failing container can be isolated by name, apply the above fixes only to that container's spec fields and verify via kubectl describe that no sibling container's lifecycle is altered.

Prove the fix

1. 01Run kubectl get pod <pod> -n <namespace> -o wide and confirm RESTART COUNT remains at the post-fix value across at least two poll intervals, with the pod phase stable as Running and no transition back to CrashLoopBackOff.
2. 02Run kubectl describe pod <pod> -n <namespace> and confirm that the Events section contains no new entries with the reason Back-off restarting failed container or ProbeFailed after the fix was rolled out.
3. 03Run kubectl logs <pod> -n <namespace> --previous --tail=50 and confirm that the most recent container instance reached a stable steady-state log pattern rather than terminating abruptly or freezing before its readiness endpoint responded.
4. 04Issue a read-only readiness check from outside the pod against the service's ClusterIP on the documented port and observe a successful response, which proves the readinessProbe contract holds and not only that the process is alive.
5. 05For probe fixes, re-run the probe handler manually inside a debug container using the same command, path, or socket as the spec, and confirm a successful response within timeoutSeconds so the next restart cannot recur from the same cause.

Prevention and next steps

• •Adopt a startupProbe for any container whose cold-start time varies with data size, cache warming, or downstream connectivity, and gate livenessProbe and readinessProbe behind it so the kubelet cannot kill the process before startup completes.
• •Define requests and limits from observed p99 memory and CPU footprints rather than from estimates, and review them whenever the application introduces a new dependency that loads at boot.
• •Keep probe handlers aligned with a stable contract: an HTTP path that always exists, a TCP port that is bound before the probe starts, or an exec command whose exit code is governed by a single readiness signal.
• •Track Last State.Reason and Exit Code in dashboards so a rise in Error exits and a rise in ProbeFailed events are distinguishable, and route alerts by reason rather than by the umbrella CrashLoopBackOff phase.
• •Reference the Kubernetes application debugging guidance for the canonical field names and event reasons used in this guide, so triage language stays consistent with the upstream contract.

Safe commands and checks

kubectl get pod <pod> -n <namespace> -o wide
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous --tail=200
kubectl logs <pod> -n <namespace> -c <container> --previous --tail=200
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[?(@.name=="<container>")].lastState}'
kubectl get pod <pod> -n <namespace> -o yaml | sed -n '/livenessProbe/,/^[[:alpha:]]/p; /readinessProbe/,/^[[:alpha:]]/p; /startupProbe/,/^[[:alpha:]]/p'
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod> --sort-by=.lastTimestamp