Hands-on - Dynamic volume provisioning in Kubernetes: using NFS CSI for stateful application

A stateful application such as a database application requires persistent storage to preserve all changes to the storage layer. It also requires that the Kubernetes pod be able to bind to the same volume when it gets rescheduled on the same or on a different node. In Kubernetes, a persistent volume can be created manually or dynamically. The dynamic volume provisioning allows storage volumes to be created on-demand and automatically. The dynamic provisioning mechanism reduces administrative overhead since the storage volume will be managed automatically by a set of predefined rules..

Dynamic Volume Provisioning using NFS CSI - OpenLens dashboard

Using NFS CSI driver for Kubernetes: A properly configured NFS volume in Kubernetes can satisfy persistent volume requirements and can support a container’s moderate workload (Read/Write). A popular NFS CSI driver is “csi-driver-nfs” which supports dynamically persistent volume creation, volume snapshot, volume cloning and volume expansion (references are at the bottom).

Installing NFS CSI plugins in Kubernetes: Before installing “csi-driver-nfs” in Kubernetes, review the GitHub documentation at https://github.com/kubernetes-csi/csi-driver-nfs.

In this tutorial, we will be using Helm Package Manager to install the NFS CSI driver.

SQL Server as stateful application in Kubernetes

Step#1: Install helm package manger:

Consult the official Helm Package Manger install process at https://helm.sh/docs/intro/install/

Login or SSH to the control plane node and then execute the following commands as root:

Switch to root:
# sudo -i

Install Helm packagfe Manager:
# curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null

# apt-get install apt-transport-https --yes

# echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list

# apt-get update

# apt-get install helm

Step#2: Using Helm to install NFS CSI:

# helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts

Search latest chart version:
# helm search repo -l csi-driver-nfs

Use the latest version of NFS CSI driver or use the specific version you like:
# helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system --version v4.5.0

Once installed, verify that NFS CSI is running on all nodes:
# kubectl get pod -n kube-system -o wide | grep nfs

Step#3: Creating a StorageClass for Dynamic Volume Provisioning:

Once we have installed NFS CSI driver in Kubernetes, the next step is to create:

1. A Storage Class (SC)
2. A Persistent Volume Claim (PVC)
3. A pod which will claim the PVC

To create a storage class, we need to have a NFS share somewhere in the network. If you don’t have one, then you’ll need to install and configure a NFS share and then map the NFS client (control plane) root user to the NFS server root user. In this example, we are using QNAP NFS v4.1. The required permissions have been granted to access the share as root from the Kubernetes control plane.

NFS Server IP: 192.168.0.25
NFS Share: kubedata

Create a storage class:

Following is the yaml for the storage class object. Save it as "nfs_sc.yaml".

 

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

  name: nfs-data

provisioner: nfs.csi.k8s.io

parameters:

  server: 192.168.0.25

  share: /kubedata

reclaimPolicy: Retain

allowVolumeExpansion: true

volumeBindingMode: Immediate

mountOptions:

  - hard

  - nfsvers=4.1

 

Now apply the yaml:
# kubectl apply -f nfs_sc.yaml

Example #1: A Pod using Dynamic Volume Provisioning:

When we use the dynamic volume provisioning method, we don’t need to manually create the Persistent Volume (PV) in advance. Instead, when the Persistent Volume Claim (PVC) is created, the required PV will also be automatically created and bound to the PVC. When a pod is created by referring to the PVC name, the required storage will be attached with the pod.

When creating a stateful pod, the Persistent Volume Claim (PVC) name must be provided in the specification section of the storageClassName. If we make the storage class the default storage in the Kubernetes cluster, then the storageClassName is not required. A PV will automatically be created from the default storage of Kubernetes as per the PVC and the required storage will be attached to the container.

Create a Persistent Volume Claim (PVC) yaml. Save the file as "nfs_pvc.yaml":

 

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: nfs-srvsql01-pvc

spec:

  accessModes:

    - ReadWriteOnce

  resources:

    requests:

      storage: 5Gi

  storageClassName: nfs-data

 

Apply the yaml to create the PVC (nfs-srvsql01-pvc):
# kubectl apply -f nfs-pvc.yaml

Creating a deployment and claiming the PVC:
In this step, we will create a SQL Server container:

1. Create a SQL Server deployment yaml file (sql1.yaml)
2. Create a Nodeport service to connect the SQL Server Instance from the network

The "sql1.yaml" file contains the following definition:

 

apiVersion: apps/v1

kind: Deployment

metadata:

  name: srvsql01

spec:

  replicas: 1

  strategy:

    type: Recreate  

  selector:

    matchLabels:

      app: srvsql01

  template:

    metadata:

      labels:

        app: srvsql01

    spec:

      terminationGracePeriodSeconds: 0

      hostname: srvsql01

      securityContext:

        fsGroup: 10001

      containers:

      - name: srvsql01

        image: mcr.microsoft.com/mssql/server:2022-latest

        ports:

        - containerPort: 1433

        env:

        - name: MSSQL_SA_PASSWORD

          value: "FantasticPassword"

        - name: MSSQL_PID

          value: "QQQQQ-PPPPPP-DDDDD-WWWWW-RRRRR"

        - name: ACCEPT_EULA

          value: "Y"

        - name: MSSQL_TCP_PORT

          value: "1433"

        - name: MSSQL_AGENT_ENABLED

          value: "true"  

        resources:

          requests:

            memory: 4Gi

            cpu: '2'

          limits:

            memory: 4Gi

        volumeMounts:

        - name: srvsql01-vol

          mountPath: /var/opt/mssql

          subPath: srvsql01

      volumes:

      - name: srvsql01-vol

        persistentVolumeClaim:

          claimName: nfs-srvsql01-pvc

---

apiVersion: v1

kind: Service

metadata:

  name: srvsql01-svc

spec:

  type: NodePort

  selector:

    app: srvsql01

  ports:

    - name: srvsql01

      port: 1433

      nodePort: 31433

      targetPort: 1433

      protocol: TCP

 

Create the deployment:
# kubectl apply -f sql1.yaml

Example #2: Creating StatefulSet replica using Dynamic Volume Provisioning.

A stateful deployment of a container is slightly different than a simple deployment. The basic steps are:

1. Create a service definition
2. Create a StatefulSet definition with volumeClaimTemplates

In the stateful definition, the critical part is the volumeClaimTemplates section. This is the section where we define the PersistentVolumeClaim. When a StatefulSet needs to create a pod replica, it uses the volumeClaimTemplates definition to create a PVC, and then a PV will automatically be created with the required volume for the pod.

Following is the StatefulSet definition. Save it as "sql2.yaml" and then apply it using kubectl.

# First define the service

apiVersion: v1

kind: Service

metadata:

  name: srvsql03-svc

spec:

  type: NodePort

  selector:

    app: srvsql03

  ports:

    - name: srvsql03

      port: 3433

      nodePort: 31033

      targetPort: 3433

      protocol: TCP

---

# Create the stateful replica

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: srvsql03

spec:

  replicas: 1

  selector:

    matchLabels:

      app: srvsql03

  serviceName: "srvsql03-svc"    

  template:

    metadata:

      labels:

        app: srvsql03

    spec:

      terminationGracePeriodSeconds: 10

      hostname: srvsql03

      securityContext:

        fsGroup: 10001

      containers:

      - name: srvsql03

        image: mcr.microsoft.com/mssql/server:2022-latest

        ports:

        - containerPort: 3433

        env:

        - name: MSSQL_SA_PASSWORD

          value: "YourPasswordHere"

        - name: MSSQL_PID

          value: "ABCDE-XYZXY-ZZZZZ-GGGGG-XZZZZ"

        - name: ACCEPT_EULA

          value: "Y"

        - name: MSSQL_TCP_PORT

          value: "3433"

        - name: MSSQL_AGENT_ENABLED

          value: "true"  

        resources:

          requests:

            memory: 4Gi

            cpu: '2'

          limits:

            memory: 4Gi

        volumeMounts:

        - name: nfs-srvsql03-pvc

          mountPath: /var/opt/mssql

          subPath: srvsql03

  # Dynamic volume claim goes here

  volumeClaimTemplates:

  - metadata:

      name: nfs-srvsql03-pvc

    spec:

      accessModes: [ "ReadWriteOnce" ]

      storageClassName: "nfs-data"

      resources:

        requests:

          storage: 6Gi

Create the stateful replica:
# kubectl apply -f sql2.yaml

Using NFS CSI in Kubernetes cluster

References:

Dynamic Volume Provisioning:
https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/

StatefulSets:
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

NFS CSI driver for Kubernetes:
https://github.com/kubernetes-csi/csi-driver-nfs

Kubernetes Container Storage Interface (CSI) Documentation:
https://kubernetes-csi.github.io/docs/drivers.html

Helm Package Manager:
https://helm.sh/docs/intro/install/