Kubernetes OOMKilled: correlate the limit with the process lifetime

The symptoms

• •Pod status shows a container with Last State Terminated and Reason OOMKilled in `kubectl describe pod`.
• •Container restartCount increases while the pod phase remains Running after a CrashLoopBackOff transition.
• •kubectl events for the pod report BackOff restarting failed container with a preceding OOMKilled reason.
• •Application metrics show a monotonic memory climb that crosses the limit just before the termination timestamp.
• •Steady-state working set sits well below the limit, but a specific operation (warm-up, cache fill, query) pushes it over the boundary.
• •Node-level dmesg or kubelet logs show cgroup memory throttling or kill events aligned with the container's lifecycle.

Likely causes

• •The memory limit in the pod spec is set below the process's actual working-set peak, so a normal workload operation crosses the cgroup boundary.
• •A memory leak grows resident set size over the container's lifetime until it finally exceeds a limit that was previously adequate.
• •A startup or warm-up phase (class loading, cache priming, JIT buffers) allocates more memory than the steady-state budget assumed.
• •The container is missing a memory limit, so it inherits a namespace quota or LimitRange that is tighter than the workload needs.
• •The metric being watched (container_memory_working_set_bytes) diverges from the cgroup memory limit accounting because of page cache or kernel overhead.
• •A sidecar or shared-memory volume consumes memory that counts against the same cgroup as the main process.

First ten minutes

1. 01Confirm the termination reason with `kubectl describe pod <pod> -n <namespace>` and read the Last State Terminated Reason and Exit Code fields.
2. 02List recent events for the pod with `kubectl get events --field-selector involvedObject.name=<pod> -n <namespace>` to see BackOff and OOMKilled events in order.
3. 03Identify which container in the pod was killed; OOMKilled is reported per container, not per pod.
4. 04Read the container's resource stanza from the pod spec to capture both resources.limits.memory and resources.requests.memory for the affected container.
5. 05Compare the limit value against the container's peak working_set_bytes from the metrics endpoint over the same window the kill occurred.
6. 06Check the pod's effective LimitRange and ResourceQuota in the namespace using `kubectl get limitrange,resourcequota -n <namespace> -o yaml` to rule out an inherited boundary.

Evidence to collect

• •Timestamp of the OOMKilled event from kubectl describe output and the corresponding container restart count.
• •The exact memory limit string (for example 512Mi) applied to the killed container at the time of termination.
• •container_memory_working_set_bytes and container_memory_rss samples from the 5 to 10 minutes preceding the kill.
• •Application-level heap or allocator stats for the same window, if the workload exposes them.
• •Node kubelet logs or cgroup memory.events file content showing the oom_kill counter increment for the container's cgroup path.
• •Whether any sidecar containers share the same pod cgroup and their individual memory usage at kill time.

Where to look

• •The cgroup v2 path /sys/fs/cgroup/.../<container-id>/memory.current and memory.events for the killed container on the node that hosted the pod.
• •The pod spec field spec.containers[].resources.limits.memory and spec.containers[].resources.requests.memory.
• •The namespace's LimitRange object, which can set default memory limits that override absent values in the pod spec.
• •The kubelet log on the node (typically /var/log/kubelet.log or journalctl -u kubelet) for memory pressure or eviction entries near the kill time.
• •The metrics endpoint scraped by Prometheus or the cluster's metrics-server for container_memory_working_set_bytes labeled with the pod and container.
• •kubectl describe output sections State and Conditions, plus Events at the bottom, for the canonical Reason string.

Diagnostic steps

1. 01If the Reason is OOMKilled and Exit Code is 137, attribute the termination to a cgroup memory kill rather than an application crash; proceed to memory correlation.
2. 02If Exit Code is 139 or 134 instead, suspect a native segfault or runtime abort and treat the OOMKilled label as misleading; inspect core dumps and runtime logs first.
3. 03Compute the ratio of peak working_set_bytes to the configured memory limit over the kill window; values consistently above 0.9 indicate the limit is the binding constraint.
4. 04Plot or sample working_set_bytes across the container's full lifetime, not just the last minute, to distinguish a steady leak from a one-shot spike tied to a specific request or job.
5. 05Compare the lifetime curve to the application's expected memory profile (constant, bounded cache, JVM heap, native buffers) to decide whether the growth is normal or pathological.
6. 06Check whether a LimitRange in the namespace is silently capping the limit you set; if so, the effective limit in describe output will differ from your manifest.
7. 07Verify whether multiple containers in the pod share a single cgroup (default behavior in cgroup v2 with Kubernetes) and whether sidecar memory counts against the main container's budget.

Common mistakes

• •Treating OOMKilled as an application bug and increasing the limit without checking working_set_bytes, which masks leaks and inflates node pressure.
• •Reading container_memory_usage_bytes instead of container_memory_working_set_bytes and concluding headroom exists when page cache is hiding real growth.
• •Assuming requests and limits are independent: exceeding the limit causes the kill even if requests were respected for scheduling.
• •Ignoring a namespace LimitRange that clamps the requested limit to a lower default, leading to repeated kills after a manifest change that appeared to take effect.
• •Restarting the pod as a remediation step without capturing pre-kill memory samples, which destroys the evidence needed to size a correct limit.
• •Conflating node-level memory pressure eviction (Reason Evicted) with container-level OOMKilled; the remediation paths differ.

Safe fixes

• •If working_set_bytes peaks just above the limit during a known warm-up, raise the limit only to a value documented in the workload's memory budget, and keep requests at or below the previous limit to preserve scheduling behavior.
• •If working_set_bytes grows monotonically across the lifetime with no plateau, treat it as a leak and roll back or patch before adjusting limits; do not raise the limit past the node's allocatable memory.
• •If a LimitRange is clamping the limit, either update the LimitRange to a value consistent with workload needs or add an explicit limit in the pod spec that exceeds the LimitRange default.
• •For JVM or similar runtimes, align the in-process heap ceiling (such as -Xmx) to leave headroom under the cgroup limit for non-heap, metaspace, threads, and native overhead.
• •For multi-container pods, sum the expected peak usage of every container in the pod and set the per-container limits so the pod total stays within node capacity and any applicable quota.
• •After any change, redeploy and observe at least one full lifetime cycle of working_set_bytes before declaring the fix stable.

Prove the fix

1. 01kubectl describe pod <pod> -n <namespace> shows no new OOMKilled entry in Last State across at least one full lifetime cycle after the change.
2. 02container_memory_working_set_bytes stays below the new limit with a documented safety margin (commonly below 80 percent of the limit) across the same window.
3. 03container_restart_count for the affected container does not increment during the observation window.
4. 04A synthetic load or replay that previously reproduced the kill completes without producing a new BackOff or OOMKilled event.
5. 05If a leak was suspected, working_set_bytes plateaus rather than trending upward across successive restarts with comparable load.

Prevention and next steps

• •Define a memory budget per workload that states steady-state working set, expected peak during warm-up, and the cgroup limit, and store it alongside the manifest.
• •Add an alert on container_memory_working_set_bytes approaching the limit (for example at 80 percent sustained) so growth is detected before a kill.
• •Track container restart count and OOMKilled event rate as SLO signals so regressions surface in dashboards rather than as incidents.
• •Review namespace LimitRange and ResourceQuota whenever a new workload is onboarded so default limits match the expected memory profile.
• •Run periodic chaos or load rehearsals that push the workload past steady state in a staging environment to validate headroom assumptions.

Safe commands and checks

kubectl describe pod <pod> -n <namespace>
kubectl get events --field-selector involvedObject.name=<pod> -n <namespace> --sort-by=.lastTimestamp
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].resources.limits.memory}{"\n"}'
kubectl get limitrange -n <namespace> -o yaml
kubectl get resourcequota -n <namespace> -o yaml
cat /sys/fs/cgroup/.../<container-id>/memory.events
kubectl logs <pod> -n <namespace> --previous --tail=200