AWS: How to Resize/Extend Persistent Volumes in Kubernetes

AWS: How to Resize/Extend Persistent Volumes in Kubernetes 

Disk Expansion in Kubernetes
Think of a situation where your Workload's Persistent Volume is almost full. How can you expand it? With Kubernetes 1.11 and above, this can now easily be done by just updating the Persistent Volume Claim storage specification.

My example in this post is tested in AWS hosted opensource Kubernetes Engine.

Let's get started!

You can only expand a PVC if its storage class’s allowVolumeExpansion field is set to true.

Enabling Volume Expansion in Storage Class
To enable the disk resizing feature, ensure that the Storage Class that your Workload's Persistent Volume Claim uses is configured with "allowVolumeExpansion: true"

[root@k8s-master manasmonitoring]# kubectl describe storageclass gp2-ebs
Name:            gp2-ebs
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"true"},"creationTimestamp":"2020-03-24T14:23:22Z","labels":{"k8s-addon":"storage-aws.addons.k8s.io"},"name":"gp2-ebs","resourceVersion":"282","selfLink":"/apis/storage.k8s.io/v1/storageclasses/gp2","uid":"7de3480c-9711-4087-a8f4-4b7c515e9d91"},"parameters":{"type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","reclaimPolicy":"Retain","volumeBindingMode":"Immediate"}
,storageclass.beta.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/aws-ebs
Parameters:            type=gp2
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Retain
VolumeBindingMode:     Immediate
Events:                <none>
[root@k8s-master manasmonitoring]#

Resizing a Persistent Volume Claim

Find the Persistent Volume Claim to resize.

[root@k8s-master ~]# kubectl get pvc  -n kube-system
NAME                                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
alertmanager                              Bound    pvc-17d71d1f-d2ea-482a-b37d-648a9f820a2a   5Gi        RWO            gp2-ebs        41h
grafanastorage-pvc                        Bound    pvc-2ee50955-db6a-4752-9c8f-429708ffc9b7   5Gi        RWO            gp2-ebs        42h
prometheus-data-prometheus-prometheus-0   Bound    pvc-a4eb87f4-dac0-4e80-952e-6b26b5bd3da0   12Gi       RWO            gp2-ebs        41h
[root@k8s-master ~]#

[root@k8s-master grafana-new-manas]# kubectl edit pvc grafanastorage-pvc -n kube-system

change 5Gi to 6Gi

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
    volume.kubernetes.io/storage-resizer: kubernetes.io/aws-ebs
  creationTimestamp: "2020-07-14T10:20:42Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: "true"
  name: grafanastorage-pvc
  namespace: kube-system
  resourceVersion: "19621218"
  selfLink: /api/v1/namespaces/kube-system/persistentvolumeclaims/grafanastorage-pvc
  uid: 2ee50955-db6a-4752-9c8f-429708ffc9b7
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 6Gi --> Here
  storageClassName: gp2-ebs
  volumeMode: Filesystem
  volumeName: pvc-2ee50955-db6a-4752-9c8f-429708ffc9b7
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 6Gi  --> Here
  phase: Bound

After successfully modifying the Persistent Volume Claim, Check its status

So the operation is based on 2 things:

1. StorageClass should be dynamic.
2. Kubernetes Version.

This procedure surely work for version 1.17 and higher as I've tested it. If you have a version Lower than this and If the solution do not work, you can do the following:

After successfully modifying the Persistent Volume Claim, Check its status

kubectl get pvc <pvc name> -o yaml

It should show you type as "type: FileSystemResizePending".

  status: "True"
    type: FileSystemResizePending
  phase: Bound

Restart the Pod
The next step is to restart the Pod as recommended by Kubernetes.

kubectl delete pod <pod name>

Then wait until the Pod starts running again.

Check the Status again:

kubectl get pvc <pvc name> -o yaml

It should show you now the changed Size.