How to test Kubernetes memory behavior before OOMKilled

The symptoms

• •Container exits with reason OOMKilled while node and pod metrics show available memory, suggesting the limit was reached before the workload visibly degraded.
• •memory.working_set_bytes plateaus near the configured limit while RSS in the container stays lower, indicating reclaimable cache or page cache pressure.
• •Latency rises and p99 spikes appear before exit, consistent with reclaim activity rather than allocation failure.
• •No node-level OOM event in dmesg despite container termination, pointing at the cgroup boundary rather than node pressure.
• •Restart count increments rapidly after a memory.boundary test workload is applied, even though requests were sized headroom.

Likely causes

• •Working set is larger than the limit because of cache, heap fragmentation, or per-thread arenas retained by the runtime.
• •Requests sized below working set, so the scheduler never reserves enough memory and reclaim begins immediately under load.
• •QoS class is BestEffort or Burstable without a limit matching the request, so the cgroup can grow until it is killed.
• •Sporadic allocations from background goroutines, JIT, or glibc malloc arenas that push past the steady state observed in normal traffic.
• •Sidecar or shared cgroup memory accounting that aggregates usage from more than the application container.

First ten minutes

1. 01Confirm the failing pod and container names from the controller and capture the exit reason field from the most recent terminated state.
2. 02Record the configured resources.limits.memory and resources.requests.memory for the container, then record the QoS class derived from those values.
3. 03Pull the cgroup memory.events counters for the container's cgroup path to confirm a kill event tied to the limit rather than the node.
4. 04Compare memory.working_set_bytes, memory.usage_bytes, and memory.rss_bytes from the kubelet metrics endpoint over the last hour.
5. 05Note any throttling or reclaim signals in container_memory_failures_total and memory.stat's inactive_file and active_file fields before declaring OOM.

Evidence to collect

• •The terminated container's last exit status, reason, and finished-at timestamp from the pod's previous instance.
• •The container cgroup memory.events low and high counters, and memory.peak before the kill.
• •memory.working_set_bytes, memory.rss_bytes, memory.cache_bytes, and memory.failures.total over a window that spans the incident.
• •QoS class and the exact requests and limits for memory, including any LimitRange defaults applied by the namespace.
• •Node-level allocatable, available, and pressure stall information for the same window to rule out node pressure.

Where to look

• •The kubelet's read-only /metrics resource on the node, scoped to the container label set, for cgroup-backed memory metrics.
• •The container cgroup directory under the kubelet's cgroup root, specifically memory.events, memory.peak, memory.current, and memory.stat.
• •The pod status field .status.containerStatuses[*].lastState.terminated.reason, scoped to the namespace and pod name.
• •The namespace LimitRange and ResourceQuota objects, which can silently override the limits observed at runtime.
• •The events emitted on the pod and node for FailedKillPod, FailedCreate, or Evicted, which surface the boundary that fired.

Diagnostic steps

1. 01Compare the observed peak usage in memory.peak against the configured limit to decide whether the boundary was hit cleanly or exceeded by a burst.
2. 02Inspect memory.stat's total_cache and total_inactive_file to estimate reclaimable cache and separate working set from RSS.
3. 03Cross-check the QoS class against requests and limits: Guaranteed requires requests equal limits, Burstable permits a higher limit, BestEffort sets neither.
4. 04Read memory.events.high to confirm a cgroup high boundary trip versus memory.events.low for memory.low pressure reclaim signals.
5. 05Reproduce by driving a controlled allocator up to a fraction of the limit and capture the same metric set, then compare the shape of the curve.
6. 06Decide between a limit raise, a request raise, or a workload change based on whether the working set, the cache, or the burst caused the kill.

Common mistakes

• •Treating OOMKilled as a node-level event when memory.events.high is the real signal and the node has ample available memory.
• •Conflating memory.working_set_bytes with RSS, which hides cache and can lead to overprovisioning when cache is the culprit.
• •Raising the limit without verifying QoS, which can convert a BestEffort pod into a Burstable pod and change eviction behavior.
• •Ignoring LimitRange defaults that override the manifest, producing a smaller limit than what the deployment author intended.
• •Trusting average memory metrics over peak memory metrics, which hides the burst that crossed the boundary.

Safe fixes

• •Reproduce with a canary workload that allocates a fixed fraction of the limit and reports working set, peak, and RSS for comparison.
• •Set resources.requests.memory to a value at or above the observed working set, then re-measure to confirm reclaim no longer triggers at idle.
• •Tune the runtime allocator to release arenas or slabs during quiescent periods, then re-run the canary to confirm a lower steady-state RSS.
• •Isolate sidecars into their own containers with independent limits so the application cgroup reflects only the application.
• •Add a memory high watermark below the limit and observe memory.events.high to detect reclaim before a kill becomes inevitable.

Prove the fix

1. 01A canary run sustains target load for the soak window with memory.peak remaining below the limit and memory.events.high stable across the run.
2. 02The pod's terminated.reason field does not contain OOMKilled across at least N consecutive restarts under the same load profile.
3. 03memory.working_set_bytes remains above the request and below the limit with a stable margin, and memory.stat's cache fraction does not spike before exit.
4. 04A synthetic over-pressure run that exceeds the previous peak still exits with OOMKilled, demonstrating the boundary remains enforced rather than silently disabled.
5. 05QoS class is documented for the canary and matches the operational expectation, so eviction and priority semantics are preserved after the change.

Prevention and next steps

• •Track memory.peak alongside memory.working_set_bytes in dashboards so the boundary is visible before any kill occurs.
• •Run a periodic canary that exercises a fraction of the limit and emits working set and peak to a shared metric stream.
• •Enforce a namespace LimitRange that matches the application's steady-state working set to prevent silent overcommit.
• •Alert on memory.events.high increments per container rather than only on OOMKilled, so reclaim pressure is visible early.
• •Document the chosen request, limit, and QoS class together so future sizing decisions preserve the verification contract.

Safe commands and checks

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources}'
kubectl describe limitrange -n <namespace> <limitrange-name>
cat /sys/fs/cgroup/system.slice/<cgroup-path>/memory.events
cat /sys/fs/cgroup/system.slice/<cgroup-path>/memory.peak
cat /sys/fs/cgroup/system.slice/<cgroup-path>/memory.stat | grep -E 'total_cache|total_rss|total_inactive_file'
kubectl get --raw /api/v1/namespaces/<namespace>/pods/<pod-name>/metrics
kubectl top pod <pod-name> -n <namespace> --containers