A discussion came up the other day. In summary - Kubernetes local persistent volume support isn't exactly first class. We needed to ensure data persistence beyond the lifecycle of a container and didn't want to pay for an external storage solution. For the sake of this discussion a hyperconverged storage option would work best. This could be something like a Portworx, but free.

For fun, as part of this thought experiment, I will be deploying OpenEBS for Kubernetes with a LocalPV CSI plugin for ZFS. And see if I can provision some persistent volumes.

To follow along you'll need:

• A three node Kubernetes cluster, or the resources to create one. (3x 2 CPU 4GB RAM ~300GB HDD).

1. Deploy a Kubernetes cluster. You can use a cloud-based one, however the ZFS setup may require multiple volumes. For this example, I used a local cluster, where each node has three disks attached. If you want to set one up from scratch, you can create a Kubernetes Cluster on Debian 9. Note that this is an older article and you may want to install Kubernetes 1.18 instead.
2. Deploy Helm on your cluster. Follow along to install the Helm Package Manager.
3. Deploy OpenEBS on your cluster. Run helm install openebs stable/openebs --version 1.8.0. Make sure you check that the pods are up and running kubectl get pods -n openebs.

   nobby-fish-openebs-admission-server-7dd9c96fb9-9tsln      1/1     Running   0          55s
   nobby-fish-openebs-apiserver-867bcc6755-7rk76             1/1     Running   0          55s
   nobby-fish-openebs-localpv-provisioner-67c578db5c-6qbg9   1/1     Running   0          55s
   nobby-fish-openebs-ndm-gjnld                              1/1     Running   0          55s
   nobby-fish-openebs-ndm-mnzqb                              1/1     Running   0          55s
   nobby-fish-openebs-ndm-operator-5966cc94fc-6vqdt          1/1     Running   1          55s
   nobby-fish-openebs-provisioner-7bfb6ddff8-5fv2z           1/1     Running   0          55s
   nobby-fish-openebs-snapshot-operator-56676c5db-9bvsd      2/2     Running   0          55s

4. Choose a storage engine. Depending on which Container Attached Storage Engine (CAS) you use you may wish to install other dependencies. OpenEBS comes with three storage engines:
   • Jiva: for applications with small data sets that require replication
   • cStore: for applications where performance is more important, replication, snapshots and clones are supported
   • LocalPV: high speed but with no replication, backup and restore is supported

     Note that for Jiva and cStore you will need to deploy iSCSI on your cluster nodes. Either use Ansible or use sudo apt install open-iscsi. Make sure the service is running with systemctl status iscsid.

5. Deploy ZFS on the cluster nodes. This requires apt install zfsutils-linux. This package is part of debian contrib and may take some time to build from source. Make sure the zfs services are running. You may want to check with systemctl status zfs-mount.service and systemctl status zfs-share.service. Then create a zpool on each one of the nodes. I ended up doing this as a mirror of /dev/sdb and /dev/sdc, to keep it simple.

   zpool create zfspv-pool mirror /dev/sdb /dev/sdc

6. Install the OpenEBS ZFS CSI plugin. Check that the pods are running after this completes.

   kubectl apply -f https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/zfs-operator.yaml
   kubectl get pods -n kube-system -l role=openebs-zfs

   openebs-zfs-controller-0   7/7     Running   0          1m37s
   openebs-zfs-node-25cgq     2/2     Running   0          1m37s
   openebs-zfs-node-rfktq     2/2     Running   0          1m37s

7. Create a storage class and persistent volume claim.

   $ cat sc.yaml
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
   name: zfs-sc
   allowVolumeExpansion: true
   parameters:
   poolname: "zfspv-pool"
   provisioner: zfs.csi.openebs.io

   $ cat pvc.yaml
   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
   name: zfs-pvc
   spec:
   storageClassName: zfs-sc
   accessModes:
    - ReadWriteOnce
   resources:
    requests:
      storage: 1Gi

8. Apply your storage class and persistent volume claim and check the results.

   kubectl apply -f sc.yaml
   kubectl apply -f pvc.yaml
   kubectl get pvc

   zfs-pvc   Bound    pvc-dc1b5bc2-732b-11ea-b800-000c2982ad3b   1Gi        RWO            zfs-sc         26s

That's it. We've now deployed ZFS on Kubernetes. Now the next step would be to run benchmarks to compare the performance of various ZFS configurations in combination with containers, and to check out all the other available OpenEBS options, especially those that allow to quickly recover and migrate existing persistent volumes to new containers.