What this usually means
The PVC's storage class either doesn't exist, lacks a provisioner, or the provisioner is failing. Alternatively, if the storage class has volumeBindingMode: WaitForFirstConsumer, the PVC won't bind until a pod uses it. Also check that the requested storage size doesn't exceed available PV capacity and that the access modes match.
The first ten minutes — establish facts before touching code.
- 1kubectl get pvc <name> -o wide — check StorageClass and status
- 2kubectl describe pvc <name> — look for Events at the bottom
- 3kubectl get storageclass — verify the SC exists
- 4kubectl describe storageclass <sc-name> — check provisioner and parameters
- 5kubectl get pv — see if any PVs could satisfy the claim
- 6kubectl get pods --all-namespaces | grep <provisioner-pod> — check provisioner pod health
The specific files, logs, configs, and dashboards that usually own this bug.
- searchkubectl describe pvc <name> — Events section for provisioner errors
- searchkubectl describe storageclass <name> — provisioner field and parameters
- searchkubectl logs -n kube-system <provisioner-pod> — provisioner logs
- searchkubectl get events --all-namespaces --sort-by=.metadata.creationTimestamp
- searchCloud provider console (e.g., AWS EBS, GCE PD) — check volume limits
Practical causes, not theory. These are the things you will actually find.
- warningStorageClass does not exist or is misspelled in PVC
- warningStorageClass provisioner is not installed (e.g., missing CSI driver)
- warningProvisioner pod is crashing or has insufficient permissions
- warningvolumeBindingMode: WaitForFirstConsumer — no pod has referenced the PVC yet
- warningRequested storage size exceeds available quota or capacity
- warningAccess modes mismatch (e.g., PVC uses ReadWriteOnce but only ReadOnlyMany PVs exist)
Concrete fix directions. Pick the one that matches your root cause.
- buildCreate the missing StorageClass with correct provisioner name
- buildInstall or fix the CSI driver / provisioner (e.g., ebs-csi-controller for AWS)
- buildChange volumeBindingMode to Immediate or create a pod that uses the PVC
- buildReduce the requested storage size or create a larger PV
- buildMatch access modes: change PVC or create PV with compatible access modes
A fix you cannot prove is a guess. Close the loop.
- verifiedkubectl get pvc <name> — status changes to Bound
- verifiedkubectl get pv — a new PV appears bound to the PVC
- verifiedProvisioner logs show successful volume creation (e.g., 'volume created successfully')
- verifiedPod referencing the PVC starts running (kubectl get pods <pod-name>)
- verifiedCloud console shows a new disk/volume of requested size
Things that make this bug worse or harder to find.
- warningDon't delete and recreate the PVC without checking the StorageClass first
- warningDon't assume the provisioner is running — check its pod status
- warningDon't ignore WaitForFirstConsumer — it's intentional, not a bug
- warningDon't set storageClassName: "" (empty string) unless you have a default SC
- warningDon't forget that some provisioners require node labels or topology constraints
The Missing EBS CSI Driver
Timeline
- 10:00Team deploys a StatefulSet with PVC claiming 10Gi from 'gp2' SC
- 10:02StatefulSet pods stuck in ContainerCreating, PVC shows Pending
- 10:05kubectl describe pvc shows no events, StorageClass 'gp2' exists
- 10:10Check provisioner: kubectl describe sc gp2 shows provisioner: ebs.csi.aws.com
- 10:12kubectl get pods -n kube-system | grep ebs — no ebs-csi pods found
- 10:15Helm list shows ebs-csi-driver not deployed; team had skipped it
- 10:20Install ebs-csi-driver via Helm with correct IAM role
- 10:25PVC transitions to Bound, StatefulSet pods start running
I was on-call when the team reported that a new StatefulSet was stuck. The PVC for each replica was Pending. My first instinct was to check the StorageClass — it existed, named 'gp2'. But when I described it, I noticed the provisioner was 'ebs.csi.aws.com'. That should be fine, but I couldn't see any provisioner errors in the PVC events.
I then checked for the CSI driver: kubectl get pods -n kube-system | grep ebs returned nothing. That was the root cause — the ebs-csi-driver was never installed on this new EKS cluster. The previous cluster had it as part of the EKS add-on, but this one was created via eksctl without it.
We installed the driver using Helm with the right IAM role for the controller. Within minutes, the PVCs bound and the StatefulSet came up. Lessons: always verify provisioner pods exist before debugging PVCs, and include the CSI driver in your cluster creation automation.
Root cause
EBS CSI driver (ebs-csi-controller) not deployed in the cluster, so the provisioner could not create volumes.
The fix
Install the ebs-csi-driver Helm chart with appropriate IAM role for the controller service account.
The lesson
Always check that the provisioner referenced in the StorageClass is actually running in the cluster before diving into other causes.
The StorageClass object defines the provisioner (e.g., ebs.csi.aws.com, kubernetes.io/gce-pd) and parameters like type, replication, etc. When a PVC references a StorageClass, the Kubernetes controller manager passes the request to the provisioner. If the provisioner is not installed or misconfigured, the PVC stays Pending.
Check the provisioner name exactly: kubectl describe sc <name> | grep Provisioner. Then verify that the corresponding CSI driver or in-tree plugin is deployed. For example, for AWS EBS, look for pods with 'ebs-csi' in kube-system. For GCE PD, the in-tree plugin is built into kube-controller-manager, but CSI driver is also common.
When volumeBindingMode is WaitForFirstConsumer, the PV will not be provisioned until a pod that uses the PVC is scheduled. This is common with multi-zone clusters to ensure the volume is created in the same zone as the pod. If you describe the PVC and see 'waiting for first consumer', that's expected — deploy the pod first.
If you want immediate binding, change the StorageClass to Immediate. But be aware that the volume might be created in a zone where no pod runs, causing cross-zone latency or attachment failures. Also, check if the pod has nodeSelector or topology constraints that match the provisioned volume.
Even with dynamic provisioning, the provisioner might reject requests if the requested size is outside its allowed range. For example, some CSI drivers require a minimum size (e.g., 1Gi for EBS). Check the provisioner's documentation for size limits.
Access modes must match between PVC and PV. Common modes: ReadWriteOnce (RWO), ReadOnlyMany (ROX), ReadWriteMany (RWX). A PVC requesting RWO will not bind to an existing PV with ROX. If using static provisioning (existing PV), ensure the PV has the correct access mode and enough capacity. Also, a PVC with storageClassName: "" (empty) will only bind to PVs with the same empty storage class (i.e., no class).
The most informative source is the provisioner's logs. For CSI drivers, find the controller pod: kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver (or similar). Then: kubectl logs <pod-name> -n kube-system -c <container> (often csi-provisioner or ebs-plugin). Look for lines with 'provision' or 'error'.
Also check the Kubernetes events: kubectl get events --all-namespaces --sort-by=.metadata.creationTimestamp | grep <pvc-name>. Events often contain the exact error message from the provisioner, e.g., 'failed to provision volume with StorageClass: ...'. This is faster than scanning logs.
Frequently asked questions
Why is my PVC stuck in Pending even though the StorageClass exists?
The StorageClass exists but its provisioner may not be deployed. Run kubectl describe sc <name> to see the provisioner name, then check if the corresponding CSI driver pods are running in kube-system. Also check if volumeBindingMode is WaitForFirstConsumer and no pod has referenced the PVC yet.
What does 'waiting for first consumer to be created before binding' mean?
It means the StorageClass has volumeBindingMode set to WaitForFirstConsumer. The PVC will not be bound until a pod that uses it is scheduled. This is normal behavior for multi-zone clusters. Create a pod that consumes the PVC, and it will bind automatically.
How do I fix a PVC that is stuck with 'Failed to provision volume'?
Check the provisioner logs for the exact error. Common causes: insufficient IAM permissions (cloud), quota exceeded, or driver configuration issues. Resolve the error, then delete and recreate the PVC, or the provisioner may retry automatically.
Can I force bind a PVC to a specific PV?
Yes, if you have an existing PV that matches the PVC's storage class, size, and access modes. You can edit the PV to set claimRef to the PVC's namespace and name. But dynamic provisioning is preferred. Forcing a bind is a workaround, not a fix.