Kubernetes Pod Stuck in Pending: Debugging Resource and Scheduling Failures

The scheduler emits events with specific reasons. Common ones include: `Insufficient memory` (pod requests > allocatable), `Insufficient cpu`, `0/X nodes are available: X node(s) had taint {TaintKey:Value}, X node(s) didn't match pod selector`, and `waiting for a volume to be created`. Each reason narrows the search. For taints, the event lists the exact taint key/value/effect. For node selector, it won't tell you which selector failed—you have to check the pod spec. For PVC, the event will say 'persistentvolumeclaim not found' or 'volume is already used by pod(s)'. The exact text is the most actionable piece of information.

If the event says '0/4 nodes are available: 4 Insufficient cpu', then you know the sum of CPU requests across all pods on any node exceeds allocatable. Check `kubectl top nodes` and `kubectl describe node` for current usage. Sometimes the issue is a single node with a large request that can't fit elsewhere due to pod anti-affinity or even a previous pod's resources not being freed (e.g., terminated pods with graceful shutdown delays).

In cloud environments, if no node can fit the pod, the cluster autoscaler (if enabled) will attempt to add a node. During this process, the pod remains Pending. The expected behavior: within a few minutes (typically 1-5 depending on cloud provider), a new node appears and the pod schedules. However, if the autoscaler is misconfigured or hits a limit (e.g., AWS EC2 instance limit, GCP quota), the pod stays pending indefinitely. Check autoscaler logs: `kubectl logs -n kube-system deployment/cluster-autoscaler`. Look for lines like 'no node groups can accommodate the pod' or 'scale-up failed: ...'. If you see 'scale-up is in progress', wait longer.

A common pitfall: the autoscaler may not scale up if the pod has restrictive node selectors or affinities that no existing node group matches. In that case, the autoscaler will log that no node group can accommodate the pod. You may need to update the node group configuration or add a new node group with the required labels. Also, ensure the autoscaler has permissions to modify the node group (IAM roles on AWS, service accounts on GCP).

The scheduler only considers requests, not limits, when placing pods. A node's allocatable resources are its capacity minus reserved resources (system daemons, kubelet overhead). To see allocatable: `kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, allocatable: .status.allocatable}'`. The pod's requests must be ≤ allocatable on the node. If you set limits only (no requests), Kubernetes defaults requests equal to limits, which can cause unexpected scheduling failures. Always set requests explicitly.

Sometimes a node appears to have free resources but pods still fail to schedule. Check `kubectl describe node <node>` for conditions like MemoryPressure or DiskPressure—these prevent scheduling even if allocatable appears sufficient. Also, consider resource fragmentation: a node may have 4Gi memory allocatable but only in 2Gi chunks due to other pods' reservations, so a pod requesting 3Gi won't fit. Use `kubectl top node` to see actual usage, but remember that requests matter for scheduling, not usage.

A pod with a volume claim can stay Pending if the PVC is not bound. The scheduler will not schedule the pod until the PVC is bound. Check `kubectl get pvc`; if STATUS is Pending, the persistent volume claim cannot be satisfied. Use `kubectl describe pvc <name>` to see events: 'no persistent volumes available for this claim' or 'Failed to provision volume with StorageClass'. The first means no PV exists matching the claim; the second means the dynamic provisioner is failing. Check the StorageClass: `kubectl get sc` and ensure the provisioner is correct and the cloud provider is not having issues. For example, in AWS EKS, if the StorageClass uses `kubernetes.io/aws-ebs` but the CSI driver is not installed, provisioning will fail.

Another subtle issue: the PVC may be bound but the pod still pending because the volume is being used by another pod (multi-attach error). The event will say 'volume is already used by pod(s)'. In that case, you need to ensure the access mode is ReadWriteMany (RWX) if multiple pods need to write, or use a different volume. Also, if the PVC is set to `WaitForFirstConsumer`, the volume will not be provisioned until a pod is scheduled, creating a circular dependency—the pod waits for volume, volume waits for pod. The fix is to set the volume binding mode to `Immediate` or ensure the scheduler can provision the volume first (requires CSI driver support).

Complex scheduling rules can cause Pending even when resources are abundant. Pod anti-affinity with `requiredDuringSchedulingIgnoredDuringExecution` prevents two pods from being on the same node. If you have N replicas and N nodes, but each pod has anti-affinity with itself, the scheduler can place only one per node—works. But if you have more replicas than nodes, the extra pods will stay Pending. The event will say 'X node(s) didn't match pod anti-affinity rules'.

Similarly, topology spread constraints can require pods to be spread across zones or hosts. If you have 3 zones but only 2 zones have nodes, or if the constraints are too restrictive (e.g., maxSkew=1 with more replicas than nodes), pods will pend. Check the pod YAML for `topologySpreadConstraints` and `podAntiAffinity`. The event message often includes the specific constraint that failed. To fix, you can relax the constraints (e.g., change `requiredDuringScheduling` to `preferredDuringScheduling`) or add more nodes/zones.