Longhorn is a stable open source Kubernetes distributed block storage that can be used in production. Configuring longhorn is not very complicated, but there are a few caveats to consider when using it with MicroK8S on an Ubuntu 20.04 box
This guide assumes you have set up your kubernetes cluster on a single node as per our previous guide.
After that, we can proceed as follows to set up Longhorn
1.Install nfs-common and open-iscsi
1 | root@vmi663745:~# apt install nfs-common open-iscsi -y |
2.Start and Enable nfs-common
1 2 | root@vmi663745:~# systemctl start nfs-commonroot@vmi663745:~# systemctl enable nfs-common |
In case you encounter an error that nfs-common is masked, please check the Troubleshoot section of this guide.
2. Start and Enable open-iscsi
1 | root@vmi663745:~# systemctl start open-iscsi; systemctl enable open-iscsi |
NOTE: For open-iscsi service you may first need to create an iscsi target – client setup for the service to start. If you start the service, and it fails with no errors, please check our guide on how to make it work. Ideally, if you restart open-iscsi and when checking status you get similar output as below, you need to check the guide to successfully start this service
1 2 3 4 5 6 7 | root@vmi663745:~# systemctl status open-iscsi● open-iscsi.service - Login to default iSCSI targets Loaded: loaded (/lib/systemd/system/open-iscsi.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Mon 2021-10-11 16:08:13 CEST; 2min 20s ago ├─ ConditionDirectoryNotEmpty=|/etc/iscsi/nodes was not met └─ ConditionDirectoryNotEmpty=|/sys/class/iscsi_session was not met |
3. Install Longhorn
There are 3 ways to install longhorn as per Longhorn Docs. In our case, we will install it via Helm Chart
- Add the Longhorn Helm repository:
1 | root@vmi663745:~# helm repo add longhorn https://charts.longhorn.io |
- Update Helm repo
1 | root@vmi663745:~# helm repo update |
- Create a namespace and install Longhorn
1 2 | root@vmi663745:~# kubectl create namespace longhorn-systemroot@vmi663745:~# helm install longhorn longhorn/longhorn --namespace longhorn-system |
Ideally, Longhorn should start all pods but often times – for reasons I have not yet know – it fails to successfully deploy longhorn-driver and thus several pods can’t work yet. Check the Troubleshooting issue no. 3 below for a fix
4. Set Longhorn as the default storage class
1 | root@vmi663745:~# kubectl patch storageclass longhorn -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' |
Since I will not be using the microk8s-hostpath storage provider, I can proceed to remove it
1 | root@vmi663745:~# microk8s.disable storage |
Troubleshoot
1.nfs-common service is masked
Error:
Failed to start nfs-common.service: Unit nfs-common.service is masked.
Symptoms:
Error occurs when you run systemctl start nfs-common for the first time after installation
Solution:
- Check if the nfs-common.service file exists as below and points to /dev/null
1 2 | root@vmi663745:~# ls -l /lib/systemd/system/nfs-common.servicelrwxrwxrwx 1 root root 9 May 24 22:51 /lib/systemd/system/nfs-common.service -> /dev/null |
- Remove the file
1 | root@vmi663745:~# rm -f /lib/systemd/system/nfs-common.service |
- Reload daemons then restart nfs-common
1 | root@vmi663745:~# systemctl daemon-reload |
1 | root@vmi663745:~# systemctl start nfs-common |
root@vmi663745:~# systemctl status nfs-common
● nfs-common.service - LSB: NFS support files common to client and server
Loaded: loaded (/etc/init.d/nfs-common; generated)
Active: active (running) since Mon 2021-10-11 15:23:16 CEST; 15s ago
Docs: man:systemd-sysv-generator(8)
Process: 27904 ExecStart=/etc/init.d/nfs-common start (code=exited, status=0/SUCCESS)
Tasks: 2 (limit: 9485)
Memory: 55.9M
CGroup: /system.slice/nfs-common.service
├─27926 /sbin/rpc.statd
└─27941 /usr/sbin/rpc.idmapd
#########- Enable nfs-common
1 | root@vmi663745:~# systemctl enable nfs-common |
2.MicroK8S cluster does not install packages from Helm
Error:
INSTALLATION FAILED: Kubernetes cluster unreachable: Get “http://localhost:8080/version?timeout=32s”: dial tcp 127.0.0.1:8080: connect: connection refused
Solution:
Set the kubeconfig file
1 | root@vmi663745:~# microk8s.kubectl config view --raw > ~/.kube/config |
3. Longhorn driver does not start
Issue:
After successfully installing Longhorn via kubectl or Helm method, the longhorn-driver pod starts but keeps crashing
Error logged under pod description is as below
time=”2021-09-23T13:15:38Z” level=error msg=”failed to get arg root-dir. Need to specify \”–kubelet-root-dir\” in your Longhorn deployment yaml.: failed to get kubelet root dir, no related proc for root-dir detection, error out”
time=”2021-09-23T13:15:38Z” level=fatal msg=”Error deploying driver: failed to start CSI driver: failed to get arg root-dir. Need to specify \”–kubelet-root-dir\” in your Longhorn deployment yaml.: failed to get kubelet root dir, no related proc for root-dir detection, error out”
Solution:
A workaround I found in Github Issues is running the command below
1 | root@vmi663745:~# curl https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml | sed -e 's/#- name: KUBELET_ROOT_DIR/- name: KUBELET_ROOT_DIR/g' -e 's$# value: /var/lib/rancher/k3s/agent/kubelet$ value: /var/lib/kubelet$g' | kubectl apply -f - |
After that, longhorn-driver runs and other related pods also are created and run successfully
