Every Kubernetes storage tutorial, including our own CKA PV, PVC and StorageClass guide and the CKAD persistent and ephemeral volumes guide, ends at the same moment: the PersistentVolumeClaim reports Bound, the Pod mounts it, the file is written and read back. That is the first half of a volume’s life. The exams, and real clusters, spend a lot of time in the second half.
The second half is where the claim is already Bound and somebody needs to change it. The database outgrew its disk and the claim must get bigger without a redeploy. The team wants more IOPS on the same volume. Someone needs a copy of production data in a staging namespace, or a point-in-time snapshot before a risky migration. A StatefulSet was deleted with Retain on its volumes and the data must be re-attached to a new claim. A node was decommissioned and the volume’s access mode needs to be tightened to ReadWriteOncePod. A PV with Delete policy was removed before its PVC and the cloud disk is still there, costing money.
Each of those operations has its own controller, its own status fields and its own failure modes, and almost none of them appear in the beginner material. This post covers them at the depth the CKA storage domain, the CKAD application-design domain and the KCSA storage-security questions actually test, using the Kubernetes 1.37 documentation and the storage API reference as the source. The security consequences of reclaim policies, finalizers and clones are owned by our KCSA storage security guide and are cross-referenced here rather than retaught.
The mental model: who acts on a Bound claim
Once a PVC is Bound, four different actors can still change it, and knowing which one is responsible tells you where to look when something stalls.
| Operation | Trigger | Control-plane actor | Node actor | Where status appears |
|---|---|---|---|---|
| Expansion | Edit spec.resources.requests.storage upward | external-resizer sidecar calls CSI ControllerExpandVolume | kubelet calls CSI NodeExpandVolume and grows the filesystem | PVC status.conditions, status.allocatedResources, status.allocatedResourceStatuses, status.capacity |
| Attribute change | Edit spec.volumeAttributesClassName | external-resizer sidecar calls CSI ControllerModifyVolume | none | PVC status.currentVolumeAttributesClassName, status.modifyVolumeStatus |
| Snapshot | Create a VolumeSnapshot | snapshot controller creates VolumeSnapshotContent; csi-snapshotter sidecar calls CSI CreateSnapshot | none | VolumeSnapshot.status.readyToUse, boundVolumeSnapshotContentName, restoreSize |
| Restore or clone | Create a new PVC with dataSource | external-provisioner calls CSI CreateVolume with a content source | none | New PVC phase |
| Populate | Create a new PVC with dataSourceRef to a custom kind | the matching volume populator controller | none | New PVC phase, populator events |
| Re-bind | Delete PVC, edit PV claimRef, create PVC with volumeName | PV controller in kube-controller-manager | none | PV phase Released → Available → Bound |
The pattern that explains most exam scenarios: control-plane steps happen without a Pod; node steps need the volume mounted on a node, which means a running Pod. Expansion of a filesystem volume is the only operation in the table with a node step, and that is exactly why “my PVC resize is stuck” is the most common storage ticket.
Volume expansion, end to end
The prerequisites
Expansion is stable since Kubernetes 1.24 and enabled by default, but three things must all be true for a given claim:
- The claim’s StorageClass has
allowVolumeExpansion: true. The default is false. A StorageClass is otherwise immutable in itsparameters, butallowVolumeExpansioncan be flipped on an existing class. - The volume is a CSI volume whose driver implements the expansion capability. The documentation lists
csi(including migrated in-tree types), plus the deprecatedflexVolumeandportworxVolume. If you provisioned a PV by hand with no CSI driver behind it (ahostPathorlocalPV on an exam cluster), there is nothing to expand: editing the PV’scapacitydirectly changes the number in the API and nothing else. - You edit the PVC, not the PV, and only upward. Kubernetes does not support shrinking. Editing a PV’s capacity by hand is actively harmful: if the PV and PVC sizes then match, the controller believes no resize is needed and skips the storage operation.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-expandable
provisioner: ebs.csi.aws.com
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
Then the whole user-facing operation is one patch:
kubectl patch pvc data-db-0 -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
The two-phase mechanics
What happens next is two separate phases carried out by two separate components, and the PVC status tells you which phase you are in.
Phase 1, controller expansion. The external-resizer sidecar running alongside the CSI controller plugin notices that spec.resources.requests.storage exceeds status.capacity.storage. It records the request in status.allocatedResources and sets status.allocatedResourceStatuses.storage to ControllerResizeInProgress, then calls the driver’s ControllerExpandVolume, which grows the backend disk. The PVC carries a condition of type Resizing while this runs. If the driver reports that no node-side work is needed (raw block volumes, or backends that resize the filesystem themselves), the capacity is updated and the operation is complete.
Phase 2, node expansion. For a filesystem volume, the backend disk is now bigger but the filesystem on it is not. The resizer sets allocatedResourceStatuses.storage to NodeResizePending and the PVC gets a condition of type FileSystemResizePending. The kubelet on the node where the volume is mounted calls NodeExpandVolume, sets NodeResizeInProgress, and grows the filesystem. Only XFS, ext3 and ext4 are supported. When it finishes, status.capacity.storage is updated to the new size and the status entry is cleared.
The documentation’s phrasing of the node step is the key exam fact: file system expansion is performed “when a new Pod using the PersistentVolumeClaim in ReadWrite mode is started”, or while a Pod is running if the filesystem supports online expansion, which the common ones do. Since 1.24 an in-use PVC is expanded without deleting or recreating the Pod. But the volume must be attached and mounted somewhere. A PVC with no Pod will finish phase 1, sit in FileSystemResizePending indefinitely, and report the old capacity until a Pod mounts it. That is not a failure; it is the controller waiting for a kubelet.
kubectl get pvc data-db-0 -o jsonpath='{.status.capacity.storage}{"\n"}{.status.allocatedResources}{"\n"}{.status.allocatedResourceStatuses}{"\n"}{.status.conditions[*].type}{"\n"}'
A typical mid-flight read:
10Gi
map[storage:20Gi]
map[storage:NodeResizePending]
FileSystemResizePending
The reading is unambiguous: the backend is already 20Gi, the filesystem is still 10Gi, and the next kubelet that mounts this volume will finish the job.
The status vocabulary
The allocatedResourceStatuses field is a map keyed by resource name (storage for the built-in case), and the API reference defines these values:
| Value | Meaning |
|---|---|
ControllerResizeInProgress | The resize controller has started resizing the volume in the control plane. |
ControllerResizeFailed | The controller-side resize failed with a terminal error. |
NodeResizePending | Controller resize finished; further resizing is needed on the node. |
NodeResizeInProgress | The kubelet has started resizing the volume. |
NodeResizeFailed | The kubelet resize failed with a terminal error. Transient errors do not set this. |
When the map is empty, no resize is in progress. The companion field allocatedResources records the size that was requested and may be larger than the real capacity while the operation runs; the documentation notes that for storage quota, the larger of allocatedResources and spec.resources is charged, so a failed oversize request keeps consuming quota until it is lowered.
Recovering from a failed expansion
Until recently, requesting a size the backend could not provide (beyond the disk type’s maximum, beyond quota, beyond a free-tier limit on a cloud account) was a trap: the controller retried forever and you could not lower the request because shrinking is forbidden. The RecoverVolumeExpansionFailure feature gate changed that. It went alpha in 1.23, beta and enabled by default in 1.32, and stable and locked on in 1.34.
With it, recovery is one edit: lower spec.resources.requests.storage to a value that is still greater than status.capacity but smaller than the failed request. The resizer retries with the new target, and allocatedResources is lowered once no expansion is in progress and the actual capacity is at or below the new request. The documentation is careful to say the new value must exceed the current capacity; you still cannot go below what the volume actually is.
# Requested 500Gi, backend maxed at 256Gi, status shows ControllerResizeFailed.
kubectl patch pvc data-db-0 -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'
kubectl get events --field-selector involvedObject.name=data-db-0 -w
The documentation also keeps the pre-1.34 manual procedure for clusters without the feature, and it is worth knowing because it is the same claimRef dance used for re-binding later in this post: set the PV reclaim policy to Retain, delete the PVC, remove claimRef from the PV, recreate the PVC with the smaller size and volumeName pointing at the PV, then restore the reclaim policy.
Expansion and StatefulSets
The one place expansion still surprises people is volumeClaimTemplates. The template in a StatefulSet is immutable, so you cannot bump the size there; you patch each PVC that the StatefulSet created (data-db-0, data-db-1, and so on) individually, and they expand independently while the Pods keep running. The template stays at the old size, which only matters for new replicas created later, and the documented workaround is to delete the StatefulSet with --cascade=orphan and recreate it with the larger template. The ordering and identity guarantees that make this safe are covered in our StatefulSets guide.
VolumeAttributesClass: changing a volume’s class without copying it
Expansion changes one number. Everything else about a provisioned volume (its IOPS, its throughput, its performance tier) used to be frozen at provisioning time because it came from StorageClass parameters, and storageClassName on a PVC is immutable. VolumeAttributesClass is the mechanism that makes those attributes mutable. It went alpha in 1.29, beta (off by default) in 1.31, stable in 1.34, and locked on from 1.36.
A VolumeAttributesClass is a cluster-scoped object in storage.k8s.io/v1 with two fields that matter: driverName, which must be a CSI driver that implements the ModifyVolume capability, and parameters, which are driver-specific and immutable once the class exists (at most 512 parameters, 256 KiB total, the same limits as a StorageClass).
apiVersion: storage.k8s.io/v1
kind: VolumeAttributesClass
metadata:
name: silver
driverName: csi.storage.example.com
parameters:
iops: "3000"
throughput: "50Mi"
---
apiVersion: storage.k8s.io/v1
kind: VolumeAttributesClass
metadata:
name: gold
driverName: csi.storage.example.com
parameters:
iops: "10000"
throughput: "250Mi"
A PVC opts in with spec.volumeAttributesClassName, which, unlike storageClassName, can be changed after creation. Setting it at creation time passes the parameters to CreateVolume; changing it later makes the external-resizer call ControllerModifyVolume on the existing volume. No data is copied and no Pod is restarted.
kubectl patch pvc data-db-0 -p '{"spec":{"volumeAttributesClassName":"gold"}}'
kubectl get pvc data-db-0 -o jsonpath='{.status.currentVolumeAttributesClassName} {.status.modifyVolumeStatus}{"\n"}'
The PVC status has two fields for this. status.currentVolumeAttributesClassName is the class actually applied (unset means none). status.modifyVolumeStatus exists only while a modification is being attempted and carries targetVolumeAttributesClassName plus a status with three documented values:
modifyVolumeStatus.status | Meaning | What to do |
|---|---|---|
Pending | The PVC cannot be modified because a requirement is unmet, most commonly that the named VolumeAttributesClass does not exist. | Create the class, or fix the name. The controller proceeds when the class appears. |
InProgress | The volume is being modified. | Wait. |
Infeasible | The CSI driver rejected the request as invalid (for example, parameters the backend cannot apply to this volume type). | Specify a valid class, or reset volumeAttributesClassName to its previous value (including nil) to cancel. |
Two details carry exam weight. First, an empty string or nil volumeAttributesClassName means “no class”, and resetting to the previous value is the documented way to cancel a stuck Infeasible modification. Second, when you switch to a class that omits a parameter the old class set, whether the driver reverts that attribute to its default depends on the driver; the documentation tells you to read the driver’s documentation rather than assume.
The conceptual boundary to remember: StorageClass answers “which driver and how to provision”, VolumeAttributesClass answers “which mutable quality-of-service settings”. The first is immutable on a claim, the second is not.
Snapshots: three CRDs and two controllers
Volume snapshots have been stable since Kubernetes 1.20, but they are not core API objects. They are CustomResourceDefinitions in the snapshot.storage.k8s.io API group, and the CRDs and their controller are installed by the Kubernetes distribution or the cluster administrator, not by Kubernetes itself and not by individual CSI drivers. On a bare kubeadm cluster, kubectl get volumesnapshot returns “the server doesn’t have a resource type” until someone installs them. That alone is a plausible exam troubleshooting scenario.
The three objects mirror the PV/PVC/StorageClass trio exactly:
| Snapshot object | Scope | Analogous to | Purpose |
|---|---|---|---|
VolumeSnapshotClass | cluster | StorageClass | driver, deletionPolicy (Delete or Retain), driver-specific parameters; a default can be marked with the snapshot.storage.kubernetes.io/is-default-class annotation |
VolumeSnapshot | namespaced | PersistentVolumeClaim | A user’s request for a snapshot of a PVC (spec.source.persistentVolumeClaimName) or a binding to a pre-provisioned content (spec.source.volumeSnapshotContentName) |
VolumeSnapshotContent | cluster | PersistentVolume | The actual snapshot on the backend: spec.driver, spec.source.volumeHandle (dynamic) or spec.source.snapshotHandle (pre-provisioned), spec.volumeSnapshotRef, spec.deletionPolicy, spec.sourceVolumeMode |
Two controllers divide the work. The snapshot controller (one per cluster, installed with the CRDs) watches VolumeSnapshot and VolumeSnapshotContent objects and handles binding and the creation and deletion of content objects. The csi-snapshotter sidecar (one per CSI driver deployment) watches VolumeSnapshotContent and makes the actual CreateSnapshot and DeleteSnapshot calls to the driver. If a snapshot never becomes ready, the question is which of the two is missing or failing.
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-snapclass
driver: ebs.csi.aws.com
deletionPolicy: Delete
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: db-before-migration
namespace: prod
spec:
volumeSnapshotClassName: csi-snapclass
source:
persistentVolumeClaimName: data-db-0
kubectl -n prod get volumesnapshot db-before-migration \
-o jsonpath='{.status.readyToUse} {.status.boundVolumeSnapshotContentName} {.status.restoreSize}{"\n"}'
The lifecycle facts that generate questions:
- Binding is one-to-one, enforced by the snapshot controller, exactly like PV/PVC.
- A PVC that is the source of an in-progress snapshot is protected: if you delete it, removal is deferred until the snapshot is
readyToUseor aborted. deletionPolicylives on the content, copied from the class at creation.Deleteremoves the backend snapshot and the content object when theVolumeSnapshotis deleted;Retainkeeps both, leaving a cluster-scoped object that references real data, which is the KCSA angle covered in the storage security guide.- Volume mode conversion is blocked by default. Restoring a
Filesystemsnapshot into aBlockPVC (or vice versa) requires the administrator to annotate theVolumeSnapshotContentwithsnapshot.storage.kubernetes.io/allow-volume-mode-change: "true", because a conversion can expose raw filesystem bytes. For pre-provisioned contents the administrator must also setspec.sourceVolumeMode. - Pre-provisioned snapshots flow the other way: an administrator creates the
VolumeSnapshotContentwith asnapshotHandlefrom the backend and avolumeSnapshotRefnaming theVolumeSnapshotthat will claim it, then the user creates thatVolumeSnapshotwithsource.volumeSnapshotContentName.
Restoring, cloning and populating: the dataSource family
Creating a new PVC with content already in it is one mechanism with three sources, all expressed through two fields on the PVC spec.
Restore from a snapshot
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-db-restored
namespace: prod
spec:
storageClassName: fast-expandable
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
dataSource:
name: db-before-migration
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
The external-provisioner passes the snapshot as the volume content source to CreateVolume, and the driver creates a new volume populated from it. The request must be at least the snapshot’s restoreSize, the snapshot must be in the same namespace, and the driver must support restore.
Clone from another PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: clone-of-data-db-0
namespace: prod
spec:
storageClassName: fast-expandable
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
dataSource:
kind: PersistentVolumeClaim
name: data-db-0
Note the absence of apiGroup: a PersistentVolumeClaim is a core object, so the group is empty. The documented cloning rules:
- CSI drivers only, and only drivers that implement cloning; dynamic provisioners only.
- Source and destination must be in the same namespace.
- The source PVC must be bound and available (not in use).
- The destination may use the same StorageClass as the source or a different one, and may omit
storageClassNameto use the default. - Source and destination must have the same
volumeMode: a block clone needs a block source. - The requested size must be equal to or larger than the source’s capacity.
- The clone is a fully independent volume: you can use, snapshot, clone or delete it without reference to the source, and modifying or deleting the source does not affect the clone.
dataSource versus dataSourceRef
The dataSource field only ever accepted two kinds: VolumeSnapshot and PersistentVolumeClaim. To let any custom resource serve as a content source, Kubernetes added dataSourceRef, and the AnyVolumeDataSource feature gate that enables it went stable and locked on in 1.33. The API reference documents three differences, which are the exam content:
| Behaviour | dataSource | dataSourceRef |
|---|---|---|
| Allowed kinds | Only VolumeSnapshot and PersistentVolumeClaim | Any object in a non-core API group, plus PersistentVolumeClaim |
| Disallowed values | Silently dropped | Preserved, and an error is returned |
| Namespace | Same namespace only | Any namespace, via the namespace sub-field (alpha) |
For backwards compatibility the API server keeps the two fields in sync: when dataSourceRef.namespace is not set and only one of the two fields is filled in, the other is copied from it. If both are set they must match. When dataSourceRef.namespace is set, dataSource is left empty and must stay empty.
The practical rule: use dataSource for snapshot restore and PVC clone in the same namespace (it is what the documentation examples show and what older provisioners expect), and use dataSourceRef whenever the source is a custom kind or lives in another namespace.
Volume populators
A volume populator is a controller that watches for PVCs whose dataSourceRef names its custom kind and fills the new volume from that source, for example by downloading an image, restoring a backup from object storage, or seeding a dataset. Kubernetes ships the API and a library for writing populators; the populators themselves are third-party. When a PVC’s dataSourceRef names a kind that no installed populator or provisioner handles, the documentation says binding will not succeed, so the PVC stays Pending with no provisioner picking it up.
spec:
dataSourceRef:
apiGroup: example.storage.k8s.io
kind: ExampleDataSource
name: dataset-2026-q3
Cross-namespace sources
The namespace sub-field of dataSourceRef is gated by CrossNamespaceVolumeDataSource, which has been alpha and off by default since 1.26 with no promotion so far. When enabled, a PVC in namespace staging can name a VolumeSnapshot in namespace prod as its source, subject to a Gateway API ReferenceGrant in the source namespace permitting the reference. Know that this exists, that it is alpha, and that a cluster with the gate off ignores the field, which means the everyday answer to “copy production data into staging” is still snapshot in prod, then restore in prod, or use a populator or backup tool.
Re-binding a Released PersistentVolume
This is the most-asked second-half scenario on the CKA, because it needs no CSI driver and works on any exam cluster: a PVC was deleted, the PV had persistentVolumeReclaimPolicy: Retain, the PV is now Released, and you must attach the data to a new claim.
The reason a Released PV does not simply bind to the next matching claim is spec.claimRef. Binding is bidirectional: the PVC records spec.volumeName and the PV records spec.claimRef with the claim’s name, namespace and uid. After the PVC is deleted that claimRef still points at an object that no longer exists, and the PV controller will not hand a volume holding someone’s data to an arbitrary new claim. You have to clear the reference deliberately.
kubectl get pv pv-db-0 -o jsonpath='{.status.phase} {.spec.claimRef.name} {.spec.claimRef.uid}{"\n"}'
# Released data-db-0 7c9d...
# Option A: remove the whole claimRef; the PV becomes Available to any matching claim.
kubectl patch pv pv-db-0 --type=json -p='[{"op":"remove","path":"/spec/claimRef"}]'
# Option B (safer): keep name and namespace, clear only the uid, so only a claim
# with that exact name in that namespace can bind.
kubectl patch pv pv-db-0 -p '{"spec":{"claimRef":{"uid":""}}}'
Then create the replacement claim and pin it to the volume:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-db-0
namespace: prod
spec:
storageClassName: ""
volumeName: pv-db-0
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
Three fields in that manifest are exam traps:
volumeNamereserves the specific PV. The documentation notes that withvolumeNameset, most matching criteria are skipped, including node affinity, but the control plane still checks that storage class, access modes and requested size are compatible.storageClassName: ""(explicit empty string) is required when the PV has no class. If you omit the field entirely, theDefaultStorageClassadmission plugin assigns the cluster default and the class no longer matches, so the claim staysPending. Retroactive default assignment, stable since 1.28, makes this bite even on claims created before a default existed; the only way to opt out is the explicit empty string.- The requested size must not exceed the PV’s capacity, and the access mode must be one the PV lists. A PVC asking for 20Gi cannot bind to a 10Gi PV even with
volumeNameset.
Option B above is the documented approach in the ReadWriteOncePod migration task and the one to prefer in production: by leaving claimRef.name and claimRef.namespace in place and only blanking the uid, you reserve the PV for a specific future claim and prevent a claim in another namespace from grabbing a volume full of data. The security reasoning is spelled out in the storage security guide.
Changing a volume’s access mode: the ReadWriteOncePod migration
Access modes on a PVC are immutable, and ReadWriteOncePod (stable since 1.29, CSI volumes only) cannot be combined with any other mode. So tightening an existing ReadWriteOnce volume to ReadWriteOncePod is the re-binding procedure with one extra patch. The documented sequence:
# 1. Protect the data.
kubectl patch pv cat-pictures-pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
# 2. Stop the writer and release the claim.
kubectl scale --replicas=0 deployment cat-pictures-writer
kubectl delete pvc cat-pictures-pvc
# 3. Free the PV for a new claim with the same name and namespace.
kubectl patch pv cat-pictures-pv -p '{"spec":{"claimRef":{"uid":""}}}'
# 4. Change the mode on the PV (now unbound, so it is editable).
kubectl patch pv cat-pictures-pv -p '{"spec":{"accessModes":["ReadWriteOncePod"]}}'
# 5. Recreate the PVC with accessModes: [ReadWriteOncePod] and volumeName: cat-pictures-pv,
# then scale the writer back up.
kubectl apply -f cat-pictures-pvc.yaml
kubectl scale --replicas=1 deployment cat-pictures-writer
# 6. Restore the original reclaim policy if it was Delete.
kubectl patch pv cat-pictures-pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
The documentation adds two constraints worth memorising: only ReadWriteOnce to ReadWriteOncePod migrations are supported, and you should not make other PVC changes such as a resize until the migration is complete. The sidecar minimums for ReadWriteOncePod are csi-provisioner 3.0.0, csi-attacher 3.3.0 and csi-resizer 1.3.0.
Deletion order and the finalizers
The last second-half topic is what happens when things are deleted in the wrong order, and it is governed by three finalizers on the PersistentVolume.
| Finalizer | Added to | Purpose |
|---|---|---|
kubernetes.io/pv-protection | Every PV that is bound to a PVC | Prevents the PV object from being removed while a claim uses it; the PV sits in Terminating until the PVC goes away. (kubernetes.io/pvc-protection is the PVC-side twin that blocks PVC deletion while a Pod uses it.) |
kubernetes.io/pv-controller | Dynamically provisioned in-tree volumes (since 1.31); not added to statically provisioned in-tree volumes | Ensures the backing storage is deleted before the PV object disappears when the policy is Delete |
external-provisioner.volume.kubernetes.io/finalizer | Both dynamic and static CSI volumes (since 1.31); also replaces pv-controller on CSI-migrated volumes | Same guarantee for CSI volumes |
The last two exist because of a long-standing gap: if an administrator deleted a Delete-policy PV before its PVC, the PV object vanished immediately and the backing disk was orphaned, still billed and still holding data. The HonorPVReclaimPolicy feature gate closed that gap; it went beta on by default in 1.31 and stable in 1.33, and the gate itself has since been removed because the behaviour is now unconditional. With it, the reclaim policy is honoured regardless of deletion order: delete the PV first and it stays Terminating with the finalizer until the PVC is deleted, at which point the backend volume is removed and then the object.
kubectl get pv pv-db-0 -o jsonpath='{.metadata.finalizers}{"\n"}'
# ["kubernetes.io/pv-protection","external-provisioner.volume.kubernetes.io/finalizer"]
The exam-relevant consequence is the reverse: a PV stuck in Terminating is almost never a bug. It is either pv-protection waiting for a bound PVC to be deleted, or the provisioner finalizer waiting for the CSI driver to confirm backend deletion. Removing the finalizer by hand clears the object and leaves the real disk behind.
Two neighbouring facts round this out. The Recycle reclaim policy is deprecated, supported only by nfs and hostPath, and does nothing more than rm -rf the volume’s contents; dynamic provisioning replaced it. And the lastPhaseTransitionTime field on a PV, which records when it last changed phase, is the field a cleanup script would use to find volumes that have been Released for weeks.
In-tree plugins are gone: why every scenario above says CSI
Nearly every operation in this post requires a CSI driver, and the reason is that the in-tree volume plugins that used to ship inside Kubernetes have been removed. CSIMigration as a whole went stable in 1.25; existing PVs that reference an in-tree plugin keep working through translation to the equivalent CSI driver, but only if that driver is installed, which Kubernetes does not do for you.
| In-tree plugin | Removed in |
|---|---|
cinder (OpenStack) | 1.26 |
awsElasticBlockStore, azureDisk | 1.27 |
gcePersistentDisk | 1.28 |
azureFile, vsphereVolume | 1.30 |
portworxVolume | 1.36 |
flexVolume | Deprecated since 1.23, still present |
The StorageClass documentation’s provisioner table now shows an internal provisioner only for a dwindling set, and nfs has never had one. For exam purposes: if a question shows provisioner: kubernetes.io/aws-ebs on a current cluster, the correct diagnosis is that nothing will provision it, and the fix is the ebs.csi.aws.com provisioner with the CSI driver installed.
Diagnosis table
| Symptom | Where to look | Likely cause | Fix |
|---|---|---|---|
| PVC edit to larger size rejected by the API server | kubectl patch error | StorageClass lacks allowVolumeExpansion: true | Patch the StorageClass, retry |
PVC shows new request, capacity unchanged, condition FileSystemResizePending, status NodeResizePending | kubectl get pvc -o yaml | No Pod mounts the volume, so the kubelet cannot grow the filesystem | Start a Pod that mounts the PVC (or wait for the existing one on online-capable filesystems) |
Status ControllerResizeFailed, events mention backend limits or quota | PVC events | Requested size beyond what the backend allows | Lower the request to a value above current capacity (stable since 1.34) |
kubectl edit pv capacity changed, nothing happened | PV and PVC sizes match | PV was edited instead of the PVC | Never edit PV capacity by hand; edit the PVC |
modifyVolumeStatus.status: Pending | PVC status | Named VolumeAttributesClass does not exist | Create it or fix the name |
modifyVolumeStatus.status: Infeasible | PVC status, events | Driver rejected the parameters for this volume | Choose a valid class, or reset volumeAttributesClassName to the previous value to cancel |
kubectl get volumesnapshot: resource type not found | API discovery | Snapshot CRDs not installed | Install the CRDs and snapshot controller (distribution responsibility) |
VolumeSnapshot never readyToUse, no VolumeSnapshotContent created | snapshot controller logs | Snapshot controller missing or no VolumeSnapshotClass (and no default) | Install controller / create class |
VolumeSnapshotContent exists, snapshot never ready | csi-snapshotter sidecar logs | Driver sidecar missing or driver lacks snapshot capability | Fix the driver deployment |
Restore PVC Pending, event about volume mode | provisioner events | Filesystem snapshot restored into Block PVC (or reverse) | Annotate content with allow-volume-mode-change: "true" or match the mode |
Clone PVC Pending, size error | provisioner events | Clone request smaller than source capacity | Request at least the source size |
Clone PVC Pending, namespace error | provisioner events | Source PVC in a different namespace | Clone within the namespace; use snapshot + restore or a populator across namespaces |
PVC with dataSourceRef custom kind stays Pending forever | PVC events | No populator installed for that kind | Install the populator or use a supported source |
Released PV will not bind to a new PVC | PV spec.claimRef | Stale claimRef from the deleted claim | Clear claimRef.uid (or the whole claimRef); create PVC with volumeName |
New PVC with volumeName stays Pending, PV is Available | PVC storageClassName | Default class was injected because the field was omitted | Set storageClassName: "" explicitly |
New PVC with volumeName stays Pending, class matches | PVC size and access modes | Requested size exceeds PV capacity, or access mode not offered by PV | Lower the request / match the mode |
PV stuck Terminating | metadata.finalizers | pv-protection (PVC still bound) or provisioner finalizer waiting for backend deletion | Delete the PVC first; wait for the driver; do not strip finalizers |
provisioner: kubernetes.io/aws-ebs class provisions nothing | StorageClass | In-tree plugin removed (1.27) | Use the CSI provisioner name with the driver installed |
Lab: the second half of a volume’s life in eight steps
This lab assumes a cluster with a CSI driver that supports expansion and snapshots (any of the major cloud drivers, or the CSI hostpath driver on a kind cluster) and the snapshot CRDs installed. Steps 6 to 8 work on any cluster, including a bare kubeadm exam environment.
Step 1: Create an expandable class and a claim, and mount it.
kubectl apply -f - <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: lab-expandable
provisioner: hostpath.csi.k8s.io # replace with your driver
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lab-pvc
spec:
storageClassName: lab-expandable
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: lab-writer
spec:
containers:
- name: app
image: busybox:1.36
command: ["sh","-c","echo hello > /data/hello && sleep 1d"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: lab-pvc
EOF
kubectl wait --for=condition=Ready pod/lab-writer --timeout=120s
Step 2: Expand while in use and watch both phases.
kubectl patch pvc lab-pvc -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
kubectl get pvc lab-pvc -o jsonpath='{.status.capacity.storage} {.status.allocatedResourceStatuses} {.status.conditions[*].type}{"\n"}' -w
kubectl exec lab-writer -- df -h /data
You should see the status pass through ControllerResizeInProgress, then NodeResizePending/FileSystemResizePending, then clear, with df reporting the larger filesystem without the Pod restarting.
Step 3: Expand a claim with no Pod and observe the stall.
kubectl delete pod lab-writer
kubectl patch pvc lab-pvc -p '{"spec":{"resources":{"requests":{"storage":"3Gi"}}}}'
sleep 20
kubectl get pvc lab-pvc -o jsonpath='{.status.capacity.storage} {.status.allocatedResourceStatuses}{"\n"}'
# 2Gi map[storage:NodeResizePending] <- waits for a kubelet
kubectl apply -f lab-writer.yaml # recreate the Pod; the resize completes
Step 4: Snapshot and restore.
kubectl apply -f - <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: lab-snap
spec:
source:
persistentVolumeClaimName: lab-pvc
EOF
kubectl wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/lab-snap --timeout=120s
kubectl get volumesnapshotcontent
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lab-restored
spec:
storageClassName: lab-expandable
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 3Gi
dataSource:
name: lab-snap
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
EOF
Mount lab-restored in a throwaway Pod and confirm /data/hello exists. Note that no default VolumeSnapshotClass means the snapshot never binds; create one if the wait times out.
Step 5: Clone directly from the PVC.
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lab-clone
spec:
storageClassName: lab-expandable
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 3Gi
dataSource:
kind: PersistentVolumeClaim
name: lab-pvc
EOF
kubectl get pvc lab-clone -o jsonpath='{.spec.dataSource}{"\n"}{.spec.dataSourceRef}{"\n"}'
Observe that the API server filled in dataSourceRef to mirror dataSource. Try the clone with storage: 1Gi and read the provisioner event that rejects it.
Step 6: Retain, delete, re-bind (works anywhere).
PV=$(kubectl get pvc lab-pvc -o jsonpath='{.spec.volumeName}')
kubectl patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
kubectl delete pod lab-writer; kubectl delete pvc lab-pvc
kubectl get pv "$PV" -o jsonpath='{.status.phase} {.spec.claimRef.uid}{"\n"}' # Released <uid>
kubectl patch pv "$PV" -p '{"spec":{"claimRef":{"uid":""}}}'
kubectl get pv "$PV" -o jsonpath='{.status.phase}{"\n"}' # Available
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lab-pvc
spec:
storageClassName: lab-expandable
volumeName: $PV
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 3Gi
EOF
kubectl get pvc lab-pvc # Bound to the same PV, data intact
Repeat with storageClassName omitted on a cluster that has a default class and watch it stay Pending.
Step 7: Deletion order and finalizers.
kubectl get pv "$PV" -o jsonpath='{.metadata.finalizers}{"\n"}'
kubectl patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
kubectl delete pv "$PV" --wait=false
kubectl get pv "$PV" # Terminating, held by pv-protection
kubectl delete pvc lab-pvc # now the PV and the backend volume go
Step 8: Change attributes (needs a driver with ModifyVolume).
kubectl apply -f - <<'EOF'
apiVersion: storage.k8s.io/v1
kind: VolumeAttributesClass
metadata:
name: lab-gold
driverName: hostpath.csi.k8s.io # replace with your driver and its parameters
parameters:
iops: "10000"
EOF
kubectl patch pvc lab-restored -p '{"spec":{"volumeAttributesClassName":"lab-gold"}}'
kubectl get pvc lab-restored -o jsonpath='{.status.currentVolumeAttributesClassName} {.status.modifyVolumeStatus}{"\n"}'
Then patch to a class name that does not exist and read Pending; set it back to null and watch the status clear.
Scenario drill: cue to answer
| Scenario cue | Answer |
|---|---|
”PVC resize request rejected: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize” | Set allowVolumeExpansion: true on the StorageClass |
PVC shows 20Gi requested, 10Gi capacity, condition FileSystemResizePending, no Pod running | Schedule a Pod that mounts it; the kubelet performs the filesystem resize |
| Requested 1Ti, driver max 500Gi, resize failing in a loop | Lower the request above current capacity (1.34+ stable), or the Retain/delete/claimRef/volumeName procedure on older clusters |
| ”Shrink the PVC from 100Gi to 50Gi” | Not supported; create a smaller volume and copy data |
| Need more IOPS on an existing volume without downtime or copying | Change spec.volumeAttributesClassName to a class with higher parameters (1.34+) |
modifyVolumeStatus.status: Infeasible | Driver rejected it; set a valid class or reset to the previous class name to cancel |
Change storageClassName on an existing PVC | Immutable; recreate the PVC (snapshot/clone to migrate data) |
| “Take a point-in-time copy before a schema migration” | VolumeSnapshot with source.persistentVolumeClaimName |
kubectl get volumesnapshot returns “no resource type” | Snapshot CRDs not installed; the distribution/admin installs them |
| Snapshot content created, never ready | csi-snapshotter sidecar or driver capability problem |
| Restore a Filesystem snapshot to a Block PVC | Needs snapshot.storage.kubernetes.io/allow-volume-mode-change: "true" on the content |
| ”Copy the volume into a new PVC in the same namespace, same driver” | dataSource: {kind: PersistentVolumeClaim, name: ...}, size ≥ source |
| Clone across namespaces | Not with dataSource; dataSourceRef.namespace is alpha (CrossNamespaceVolumeDataSource) |
PVC with dataSourceRef to a CRD kind stays Pending | No volume populator for that kind |
dataSource with an unsupported kind | Silently dropped; dataSourceRef would error instead |
PV Released after PVC deletion, must reuse the data | Clear claimRef (prefer only uid), create PVC with volumeName |
PVC with volumeName Pending, PV Available, PV has no class | PVC must set storageClassName: "" explicitly |
PVC with volumeName Pending, PVC asks 20Gi, PV is 10Gi | Request must not exceed PV capacity |
| Change RWO to RWOP on an existing volume | Retain, delete PVC, clear claimRef.uid, patch PV accessModes, recreate PVC with volumeName |
PV stuck Terminating | pv-protection (PVC bound) or provisioner finalizer; delete PVC / wait; do not strip finalizers |
| Deleted a Delete-policy PV before its PVC; is the disk orphaned? | Not since HonorPVReclaimPolicy (stable 1.33); the finalizer deletes backend storage after the PVC goes |
provisioner: kubernetes.io/gce-pd on a 1.37 cluster | Removed in 1.28; use the CSI provisioner |
| StatefulSet volumes need to grow | Patch each PVC; the template is immutable (recreate with --cascade=orphan to update it) |
Recycle reclaim policy | Deprecated, rm -rf only, nfs/hostPath only |
Common mistakes
- Editing the PV’s capacity. It changes a number in the API and can prevent the real resize. Always edit the PVC.
- Expecting a resize to complete with no Pod. Phase 2 needs the kubelet. The
FileSystemResizePendingcondition is a wait, not an error. - Treating
allowVolumeExpansionas a PVC field. It is a StorageClass field, and it can be added to an existing class. - Confusing
storageClassNamewithvolumeAttributesClassName. The first is immutable and selects the provisioner; the second is mutable and selects QoS parameters on a CSI driver with ModifyVolume. - Assuming snapshots are built in. They are CRDs plus a controller that the distribution installs; a cluster can have a CSI driver and still have no snapshot API.
- Omitting
storageClassName: ""on a static re-bind. The default class gets injected and the claim never matches the class-less PV. - Clearing the whole
claimRefin a multi-tenant cluster. The volume becomesAvailableto any namespace. Clear only theuidto reserve it for the intended claim. - Cloning with a smaller request than the source. The request must be equal to or greater than the source capacity.
- Stripping finalizers from a
TerminatingPV. You remove the object and leave the backend disk orphaned, which is the exact outcome the finalizer exists to prevent. - Writing
kubernetes.io/aws-ebsor similar in a new StorageClass. Those plugins are gone; name the CSI driver.
Frequently Asked Questions
Why is my PVC resize stuck in FileSystemResizePending?
Because the backend volume has been expanded but the filesystem on it has not, and the filesystem step is done by the kubelet on the node where the volume is mounted. If no Pod mounts the PVC, nothing can perform that step. Start a Pod that uses the claim and the status clears within seconds.
Can I shrink a PersistentVolumeClaim?
No. Kubernetes does not support reducing a PVC below its current capacity. Since 1.34 you can lower a pending expansion request that failed, as long as the new value is still above the current capacity. To actually shrink, create a smaller volume and copy the data.
What is the difference between StorageClass and VolumeAttributesClass?
StorageClass chooses the provisioner and the provisioning-time parameters and is immutable on a claim. VolumeAttributesClass (stable in 1.34) describes mutable attributes such as IOPS and throughput, and a PVC’s volumeAttributesClassName can be changed after creation; the external-resizer applies it through the CSI ModifyVolume call with no data copy.
Are VolumeSnapshots part of the core Kubernetes API?
No. They are CustomResourceDefinitions in the snapshot.storage.k8s.io group (VolumeSnapshotClass, VolumeSnapshot, VolumeSnapshotContent) that the distribution or administrator installs along with the snapshot controller. Each CSI driver then runs a csi-snapshotter sidecar to perform the actual snapshot calls.
What is the difference between dataSource and dataSourceRef on a PVC?
dataSource accepts only VolumeSnapshot and PersistentVolumeClaim in the same namespace and silently drops anything else. dataSourceRef accepts any non-core kind (for volume populators) plus PVCs, returns an error for disallowed values, and has an alpha namespace field for cross-namespace sources. The API server keeps the two in sync when no namespace is specified.
How do I reuse a Released PersistentVolume?
Clear the stale spec.claimRef on the PV (preferably only its uid, so the PV remains reserved for a claim with the same name and namespace), then create a PVC that sets volumeName to the PV, requests no more than its capacity, uses a matching access mode, and sets storageClassName: "" if the PV has no class.
Can I change a PVC’s access mode?
Not directly; access modes are immutable on a PVC. The documented path for ReadWriteOnce to ReadWriteOncePod is to set the PV to Retain, delete the PVC, clear claimRef.uid, patch the PV’s accessModes, and recreate the PVC with volumeName. Only that direction of migration is supported.
Why is my PersistentVolume stuck in Terminating?
Either the kubernetes.io/pv-protection finalizer is holding it because a PVC is still bound, or the provisioner finalizer (external-provisioner.volume.kubernetes.io/finalizer for CSI, kubernetes.io/pv-controller for in-tree) is waiting for the backend storage to be deleted first, which is the HonorPVReclaimPolicy behaviour stable since 1.33. Delete the PVC or wait for the driver; removing the finalizer manually orphans the real disk.
Conclusion
A Bound PVC is not the end of a volume’s story. It can grow (edit the PVC, never the PV; the controller expands the disk and the kubelet expands the filesystem, which is why a Pod must be running), change its quality of service in place through a VolumeAttributesClass, be snapshotted through a trio of CRDs with two cooperating controllers, be restored or cloned through dataSource, be populated from a custom source through dataSourceRef, be re-bound after release by clearing claimRef and pinning volumeName with an explicit empty storageClassName, have its access mode tightened through the same procedure, and be protected from out-of-order deletion by three finalizers. Every one of those operations has a status field that tells you exactly where it is, and almost every “stuck” storage ticket is one of those fields correctly reporting that it is waiting for something.
For the CKA this is the material behind the storage domain’s troubleshooting scenarios; for the CKAD it is what the application-design questions about volume lifecycle assume; for the KCSA it is the mechanics beneath the data-lifecycle risks. If you are preparing for several of these exams at once, our KubeAstronaut certification path guide and the curriculum domain map across all five exams show where storage sits in each. For timed practice on scenarios like the ones in the drill above, where you have to read a PVC’s status fields and decide in under two minutes whether the fix is a Pod, a patch, a class or a claimRef, the KubeAstronaut mock exam bundle runs CKA, CKAD, CKS, KCNA and KCSA practice exams with per-domain breakdowns, so you can see whether storage is still where your time goes.