Kubernetes pod stays Pending: locate the scheduling constraint

The symptoms

• •Pod status remains Pending in `kubectl get pods` and never transitions to ContainerCreating, even after waiting longer than the typical admission latency.
• •`kubectl describe pod <pod>` reports a FailedScheduling event with a message starting "0/N nodes are available", where N is the cluster's node count.
• •Pod has been scheduled but no node assignment appears in the Node column; the pod has no nodeName set despite being older than the default scheduler backoff window.
• •ReplicaSet or Deployment shows desired replicas greater than ready replicas, with the unready pods all in Pending state.
• •kubectl get events for the pod's namespace shows repeated FailedScheduling warnings from the default-scheduler kube-system component without a corresponding SuccessfulBinding.

Likely causes

• •Insufficient CPU, memory, or ephemeral-storage on every node to satisfy the pod's requests; the scheduler sums requests against node allocatable and rejects placement when no node can fit.
• •NodeSelector, nodeAffinity, podAffinity, podAntiAffinity, or topologySpreadConstraints rules that no current node satisfies, often because no node carries the labeled key or the topology domain lacks peers.
• •Taints on all candidate nodes without matching tolerations on the pod, causing the scheduler to filter every node out of the feasible set.
• •PersistentVolumeClaims in Pending state because no PersistentVolume matches the requested storageClassName, accessModes, capacity, or selector, so the pod cannot start and is held by the volume binding cycle.
• •Scheduler extender, webhook, or ResourceQuota/LimitRange rejection that is not visible in the standard FailedScheduling message and only appears in the kube-scheduler logs.

First ten minutes

1. 01Run `kubectl get pod <pod> -n <namespace> -o wide` to confirm the pod, namespace, node column (should be blank), and age; record the age to distinguish a long-pending pod from one still in the initial scheduling backoff.
2. 02Run `kubectl describe pod <pod> -n <namespace>` and read the Events section at the bottom; copy the most recent FailedScheduling message verbatim, including the "0/N nodes are available" preamble and any appended reason phrases such as "insufficient cpu", "node(s) didn't match Pod's node affinity", "node(s) had taints that the pod didn't tolerate", or "node(s) didn't have free ports".
3. 03Run `kubectl get events -n <namespace> --sort-by=.lastTimestamp --field-selector involvedObject.name=<pod>` to see the full sequence of scheduler attempts and rule out transient backoff being mistaken for a permanent block.
4. 04Run `kubectl get nodes -o wide` and `kubectl describe nodes` to enumerate node capacity and the taints and labels currently applied; compare these against the failing pod's spec.
5. 05Check the kube-scheduler component log via `kubectl logs -n kube-system <scheduler-pod>` for webhook or extender denials that the per-pod describe event does not surface.
6. 06Capture the pod spec with `kubectl get pod <pod> -n <namespace> -o yaml` and isolate nodeSelector, affinity, tolerations, resources.requests, and volumeClaimTemplates for direct comparison to node state.

Evidence to collect

• •The exact FailedScheduling event message from `kubectl describe pod`, including the reason phrase appended after the node count.
• •Node list with capacity and allocatable resources from `kubectl describe nodes` and node labels from `kubectl get nodes --show-labels`.
• •Taints present on each node from `kubectl describe nodes` and the pod's tolerations from the pod spec.
• •Pod's resources.requests for cpu, memory, and ephemeral-storage, compared against node allocatable values.
• •PersistentVolumeClaim status (`kubectl get pvc -n <namespace>`) and whether it is Pending because no PV matches the storageClassName, accessModes, capacity, or selector.
• •Recent kube-scheduler log lines correlated with the pod's scheduling attempts.

Where to look

• •The kube-system boundary where the default-scheduler runs; its logs are the authoritative source for scheduling decisions and webhook rejections.
• •The kube-apiserver boundary where Pod and PersistentVolumeClaim objects are stored; their status fields and events are what kubectl reads.
• •The kubelet boundary on each node, but only after a pod has been bound; kubelet logs do not explain why the scheduler refused to bind.
• •The node object boundary where capacity, allocatable resources, labels, and taints are recorded and consulted by the scheduler's predicates.
• •The storage boundary where PersistentVolumes and StorageClasses live, because a Pending PVC will block the pod even when all node constraints are satisfied.

Diagnostic steps

1. 01Parse the FailedScheduling message to identify which filter rejected placement: resource shortfall, affinity/selector mismatch, taint without toleration, unschedulable node, or port conflict.
2. 02For resource shortfalls, sum the pod's requests across cpu, memory, and ephemeral-storage and compare to each node's allocatable values; the node with the smallest headroom relative to existing pod requests is the binding bottleneck.
3. 03For affinity or nodeSelector mismatches, run `kubectl get nodes --show-labels` and check whether any node carries the required key and value; if zero nodes match, the selector is unsatisfiable in the current cluster.
4. 04For taint issues, cross-reference each node's Taints list against the pod's tolerations; a taint effect of NoSchedule with no matching toleration filters that node out completely.
5. 05For volume-related blocks, run `kubectl get pvc -n <namespace>` and inspect `kubectl describe pvc`; a Pending PVC indicates the scheduler is waiting on the persistent volume controller, not on node placement.
6. 06For webhook or quota denials, inspect the kube-scheduler log for the pod name and look for "denied" or "rejected" messages from the configured admission or extension points.
7. 07After identifying the constraint, reproduce the scheduler's view by walking the predicates against one candidate node to confirm the filter that excluded it.

Common mistakes

• •Reading "0/N nodes are available" as a resource problem and adding nodes or raising requests, when the appended reason phrase points to affinity, taints, or volumes.
• •Restarting the kube-scheduler or deleting the pod to "unstick" it, which removes the diagnostic evidence without changing the underlying constraint; the next pod created from the same workload will Pending for the same reason.
• •Trusting the kubelet log on the target node; kubelet only logs after binding, so a Pending pod produces nothing useful there and can mislead investigation toward node health.
• •Ignoring the Pending PVC and assuming the scheduler is at fault; a pod waiting on a volume presents the same Pending surface but is a different control loop.
• •Comparing node capacity instead of node allocatable; capacity includes resources reserved by the system and is not what the scheduler sums requests against.

Safe fixes

• •If the reason phrase is "insufficient cpu/memory/ephemeral-storage", reduce the pod's resources.requests to fit within the largest node's allocatable minus current usage, or add a node with matching capacity; verify by reapplying and watching the pod transition out of Pending.
• •If the reason phrase is "node(s) didn't match Pod's node affinity/selector", either label an existing node with the required key/value or relax the nodeSelector or requiredDuringSchedulingIgnoredDuringExecution term; verify by `kubectl get pods -o wide` showing a populated Node column.
• •If the reason phrase is "node(s) had taints that the pod didn't tolerate", add a matching toleration with the appropriate effect, or remove the taint from a target node if it is operationally safe; verify by the pod binding within one scheduler cycle.
• •If the PVC is Pending, provision a matching PersistentVolume or correct the storageClassName, accessModes, capacity, or selector on the claim; verify by the PVC reaching Bound and the pod progressing to ContainerCreating.
• •If the kube-scheduler log shows a webhook or ResourceQuota denial, address the quota scope or correct the webhook configuration; verify by the scheduler emitting a binding event for the pod.
• •All fixes must be conditional on the specific reason phrase captured in step one; do not apply a fix that does not match the evidence.

Prove the fix

1. 01`kubectl get pod <pod> -n <namespace>` shows a transition from Pending to ContainerCreating within the scheduler's backoff window after the fix is applied.
2. 02`kubectl describe pod <pod> -n <namespace>` Events section contains a new SuccessfulBinding or a status that reflects a node assignment in the Pod field.
3. 03`kubectl get pods -o wide` shows the Node column populated for the previously Pending pod, confirming the scheduler bound it.
4. 04For workload-level proof, `kubectl get rs` or `kubectl get deploy` shows ready replicas equal to desired replicas, with no replicas remaining in Pending.
5. 05For regression check, re-run `kubectl get events -n <namespace> --field-selector reason=FailedScheduling` over a full scheduling cycle and confirm no new FailedScheduling events for the corrected pod template.

Prevention and next steps

• •Run `kubectl describe nodes` and capture allocatable values into workload sizing guidelines so requests stay within the smallest node's free capacity under steady state.
• •Keep node taints and pod tolerations documented together; a taint added during cluster maintenance without a corresponding toleration update will Pending every new pod of the affected workload.
• •Validate nodeSelector and affinity rules against `kubectl get nodes --show-labels` before applying manifests, so unsatisfiable selectors are caught in CI rather than at scheduling time.
• •Monitor PVC backlog with `kubectl get pvc --all-namespaces` and alert on Pending PVCs, since a stuck volume claim surfaces as a stuck pod and is easy to misattribute to the scheduler.

Safe commands and checks

kubectl get pod <pod> -n <namespace> -o wide
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp --field-selector involvedObject.name=<pod>
kubectl get nodes -o wide
kubectl describe nodes
kubectl get nodes --show-labels
kubectl get pod <pod> -n <namespace> -o yaml
kubectl get pvc -n <namespace>
kubectl logs -n kube-system <scheduler-pod>