Using ZFS Volumes with Kubernetes

last updated on 2020-03-31

A discussion came up the other day. In summary - Kubernetes local persistent volume support isn't exactly first class. We needed to ensure data persistence beyond the lifecycle of a container and didn't want to pay for an external storage solution. For the sake of this discussion a hyperconverged storage option would work best. This could be something like a Portworx, but free.

For fun, as part of this thought experiment, I will be deploying OpenEBS for Kubernetes with a LocalPV CSI plugin for ZFS. And see if I can provision some persistent volumes.

To follow along you'll need:

  • A three node Kubernetes cluster, or the resources to create one. (3x 2 CPU 4GB RAM ~300GB HDD).
  1. Deploy a Kubernetes cluster. You can use a cloud-based one, however the ZFS setup may require multiple volumes. For this example, I used a local cluster, where each node has three disks attached. If you want to set one up from scratch, you can create a Kubernetes Cluster on Debian 9. Note that this is an older article and you may want to install Kubernetes 1.18 instead.
  2. Deploy Helm on your cluster. Follow along to install the Helm Package Manager.
  3. Deploy OpenEBS on your cluster. Run helm install openebs stable/openebs --version 1.8.0. Make sure you check that the pods are up and running kubectl get pods -n openebs.
    nobby-fish-openebs-admission-server-7dd9c96fb9-9tsln      1/1     Running   0          55s
    nobby-fish-openebs-apiserver-867bcc6755-7rk76             1/1     Running   0          55s
    nobby-fish-openebs-localpv-provisioner-67c578db5c-6qbg9   1/1     Running   0          55s
    nobby-fish-openebs-ndm-gjnld                              1/1     Running   0          55s
    nobby-fish-openebs-ndm-mnzqb                              1/1     Running   0          55s
    nobby-fish-openebs-ndm-operator-5966cc94fc-6vqdt          1/1     Running   1          55s
    nobby-fish-openebs-provisioner-7bfb6ddff8-5fv2z           1/1     Running   0          55s
    nobby-fish-openebs-snapshot-operator-56676c5db-9bvsd      2/2     Running   0          55s
  4. Choose a storage engine. Depending on which Container Attached Storage Engine (CAS) you use you may wish to install other dependencies. OpenEBS comes with three storage engines:
    • Jiva: for applications with small data sets that require replication
    • cStore: for applications where performance is more important, replication, snapshots and clones are supported
    • LocalPV: high speed but with no replication, backup and restore is supported

      Note that for Jiva and cStore you will need to deploy iSCSI on your cluster nodes. Either use Ansible or use sudo apt install open-iscsi. Make sure the service is running with systemctl status iscsid.

  5. Deploy ZFS on the cluster nodes. This requires apt install zfsutils-linux. This package is part of debian contrib and may take some time to build from source. Make sure the zfs services are running. You may want to check with systemctl status zfs-mount.service and systemctl status zfs-share.service. Then create a zpool on each one of the nodes. I ended up doing this as a mirror of /dev/sdb and /dev/sdc, to keep it simple.
    zpool create zfspv-pool mirror /dev/sdb /dev/sdc
  6. Install the OpenEBS ZFS CSI plugin. Check that the pods are running after this completes.
    kubectl apply -f
    kubectl get pods -n kube-system -l role=openebs-zfs
    openebs-zfs-controller-0   7/7     Running   0          1m37s
    openebs-zfs-node-25cgq     2/2     Running   0          1m37s
    openebs-zfs-node-rfktq     2/2     Running   0          1m37s
  7. Create a storage class and persistent volume claim.
    $ cat sc.yaml
    kind: StorageClass
    name: zfs-sc
    allowVolumeExpansion: true
    poolname: "zfspv-pool"
    $ cat pvc.yaml
    kind: PersistentVolumeClaim
    apiVersion: v1
    name: zfs-pvc
    storageClassName: zfs-sc
     - ReadWriteOnce
       storage: 1Gi
  8. Apply your storage class and persistent volume claim and check the results.
    kubectl apply -f sc.yaml
    kubectl apply -f pvc.yaml
    kubectl get pvc
    zfs-pvc   Bound    pvc-dc1b5bc2-732b-11ea-b800-000c2982ad3b   1Gi        RWO            zfs-sc         26s

That's it. We've now deployed ZFS on Kubernetes. Now the next step would be to run benchmarks to compare the performance of various ZFS configurations in combination with containers, and to check out all the other available OpenEBS options, especially those that allow to quickly recover and migrate existing persistent volumes to new containers.