A discussion came up the other day. In summary - Kubernetes local persistent volume support isn't exactly first class. We needed to ensure data persistence beyond the lifecycle of a container and didn't want to pay for an external storage solution. For the sake of this discussion a hyperconverged storage option would work best. This could be something like a Portworx, but free.
For fun, as part of this thought experiment, I will be deploying OpenEBS for Kubernetes with a LocalPV CSI plugin for ZFS. And see if I can provision some persistent volumes.
To follow along you'll need:
- A three node Kubernetes cluster, or the resources to create one. (3x 2 CPU 4GB RAM ~300GB HDD).
- Deploy a Kubernetes cluster. You can use a cloud-based one, however the ZFS setup may require multiple volumes. For this example, I used a local cluster, where each node has three disks attached. If you want to set one up from scratch, you can create a Kubernetes Cluster on Debian 9. Note that this is an older article and you may want to install Kubernetes 1.18 instead.
- Deploy Helm on your cluster. Follow along to install the Helm Package Manager.
- Deploy OpenEBS on your cluster. Run
helm install openebs stable/openebs --version 1.8.0
. Make sure you check that the pods are up and runningkubectl get pods -n openebs
.nobby-fish-openebs-admission-server-7dd9c96fb9-9tsln 1/1 Running 0 55s nobby-fish-openebs-apiserver-867bcc6755-7rk76 1/1 Running 0 55s nobby-fish-openebs-localpv-provisioner-67c578db5c-6qbg9 1/1 Running 0 55s nobby-fish-openebs-ndm-gjnld 1/1 Running 0 55s nobby-fish-openebs-ndm-mnzqb 1/1 Running 0 55s nobby-fish-openebs-ndm-operator-5966cc94fc-6vqdt 1/1 Running 1 55s nobby-fish-openebs-provisioner-7bfb6ddff8-5fv2z 1/1 Running 0 55s nobby-fish-openebs-snapshot-operator-56676c5db-9bvsd 2/2 Running 0 55s
- Choose a storage engine. Depending on which Container Attached Storage Engine (CAS) you use you may wish to install other dependencies. OpenEBS comes with three storage engines:
- Jiva: for applications with small data sets that require replication
- cStore: for applications where performance is more important, replication, snapshots and clones are supported
- LocalPV: high speed but with no replication, backup and restore is supported
Note that for Jiva and cStore you will need to deploy iSCSI on your cluster nodes. Either use Ansible or use
sudo apt install open-iscsi
. Make sure the service is running withsystemctl status iscsid
.
- Deploy ZFS on the cluster nodes. This requires
apt install zfsutils-linux
. This package is part of debian contrib and may take some time to build from source. Make sure the zfs services are running. You may want to check withsystemctl status zfs-mount.service
andsystemctl status zfs-share.service
. Then create a zpool on each one of the nodes. I ended up doing this as a mirror of/dev/sdb
and/dev/sdc
, to keep it simple.zpool create zfspv-pool mirror /dev/sdb /dev/sdc
- Install the OpenEBS ZFS CSI plugin. Check that the pods are running after this completes.
kubectl apply -f https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/zfs-operator.yaml kubectl get pods -n kube-system -l role=openebs-zfs
openebs-zfs-controller-0 7/7 Running 0 1m37s openebs-zfs-node-25cgq 2/2 Running 0 1m37s openebs-zfs-node-rfktq 2/2 Running 0 1m37s
- Create a storage class and persistent volume claim.
$ cat sc.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: zfs-sc allowVolumeExpansion: true parameters: poolname: "zfspv-pool" provisioner: zfs.csi.openebs.io
$ cat pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: zfs-pvc spec: storageClassName: zfs-sc accessModes: - ReadWriteOnce resources: requests: storage: 1Gi
- Apply your storage class and persistent volume claim and check the results.
kubectl apply -f sc.yaml kubectl apply -f pvc.yaml kubectl get pvc
zfs-pvc Bound pvc-dc1b5bc2-732b-11ea-b800-000c2982ad3b 1Gi RWO zfs-sc 26s
That's it. We've now deployed ZFS on Kubernetes. Now the next step would be to run benchmarks to compare the performance of various ZFS configurations in combination with containers, and to check out all the other available OpenEBS options, especially those that allow to quickly recover and migrate existing persistent volumes to new containers.