Kubernetes on Raspberry Pi, part 3

Time for storage!

Last time we configured our Kubernetes on Raspberry Pi cluster and added the k8s Dashboard for quick-and-easy monitoring. Today we are going to add some storage to the cluster.

Technically, every node has a small SD card that provides some amount of space for writing. However, using that to store any data used for/coming from our analyses is neither practical nor recommended for several reasons. First of all, constant writing to SD cards greatly reduces their life span. Second, also related to the previous point, if one of your nodes suddenly dies, so most likely dies whatever you stored in its internal storage space. And even if it did not, it would be a hassle to transfer that to a new node. For these reasons we would like to use some sort of a more persistent storage that is independent of any particular node.

Here, we will use an external HDD/SSD drive connected via USB to one of the clusters’ nodes and exposed to all of the other nodes using NFS.

Drive configuration

In my own setup I used a Samsung SSD drive that I had lying around – feel free to use whatever you have available though – the setup described below should not be dependent on the drive type you choose.

You should start by formatting the drive to ext4 file system. As there are many tools/ways to do that and they depend a bit on your OS, I am not going to describe that here – you can find many resources out there. Come back here when your drive is ready 🙂

Once formatted, you can follow the steps below to add the storage to your cluster. This guide is heavily based on this tutorial – I will copy a chunk of it here, adding some comments at the same time. Importantly, all the credit goes to Marco Pivetta – thanks!

plug the drive into any of the nodes (I would go for the master node by default but you can choose any of the workers too)
ssh into that node:

				
					ssh pi@node_ip

switch to a root session to make typing a bit easier:

				
					sudo su

list all of the attached devices by running:

				
					fdisk -l

You should now see a list of all connected devices, among them partitions like /dev/sda1 and /dev/sda2 (that’s at least the case when connecting an SSD drive).

create a directory where the volume will be mounted on the node

				
					mkdir -p /mnt/k8s-volumes

execute the following command to add your volume to the file system table:

				
					echo “/dev/your_drive_id /mnt/k8s-volumes ext4 defaults 0 2” >> /etc/fstab

Replace your_drive_id with the ID of your drive as it appears on the list of the attached devices that we saw earlier.

mount your volume:

				
					mount -a

install NFS server:

				
					apt-get install -y nfs-kernel-server

expose the mounted drive of your network to make it accessible to the other nodes

				
					echo “/mnt/k8s-volumes 192.168.1.100/8(rw,sync,no_root_squash,no_subtree_check)” >> /etc/exports

Modify the 192.168.1.100/8 part to correspond to the range of your nodes’ IP addresses. In this example we would expose the drive to IPs in the range 192.168.1.100-192.168.1.108.

reload the table of exported NFS file systems to add our new entry:

				
					exportfs -ra

display what resources are being shared:

				
					showmount -e 127.0.0.1

You should see something like:

Export list for 127.0.0.1:
/mnt/k8s-volumes 192.168.1.100/

Test!

Let’s test whether that worked. You can do it by trying to access your network drive from any other node of the cluster.

				
					ssh pi@node_ip

switch to a root session:

				
					sudo su

create a dummy folder to mount our drive into:

				
					mkdir foo

mount the drive:

				
					mount 192.168.1.x:/mnt/k8s-volumes ./foo

Replace 192.168.1.x with the IP address of the node where you originally mounted your drive.

try to create/delete files in the foo directory – you should be able to see all the changes on the actual drive
when done with testing, unmount the resource:

				
					umount -l ./foo

Access the drive from your local machine

This part is optional. I wanted to access the drive from my Mac and here is what you need for that:

create a directory to mount the drive into:

				
					mkdir k8s-volumes

mount the volume:

				
					sudo mount -o hard,nolock,resvport 192.168.1.x:/mnt/k8s-volumes ~/k8s-volumes

Again, replace 192.168.1.x with the IP address of the node where your drive is mounted.

Persistent Volume and its Claim

There is one (or maybe two) more thing(s) that wee need to do to be able to use the storage drive we configured above. In order for the pods running in your cluster to be able to access that drive you need to create the so-called Persistent Volume and a Persistent Volume Claim.

Persistent Volume (PV) is a k8s-specific resource provisioned in the cluster. In our example, you cannot access the network drive directly from e.g. pods in the cluster. First, you need to add a volume that can then be mounted in the pod – that volume is configured as a PV. Volumes can be configured with different storage parameters and classes depending on the actual storage carrier. For more details see k8s PV documentation.

Persistent Volume Claim (PVC) is a request for storage on a specific PV. PVCs can request different storage size and access mode that will be consumed from a PV. Again, more details can be found here.

Here, we will create a sample PV and a PVC bound to it. Since we do not have any pods in our cluster yet, we will test those things later, when we deploy an actual test container.

As you already saw in the previous part (while deploying the Kubernetes Dashboard), we deploy resources to the cluster using k8s manifests. It works just the same for PVs and PVs. To deploy those we will use the following manifests:

Persistent Volume manifest:

				
					apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-persistent-volume
spec:
  capacity:
    storage: 15Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: slow
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    path: /mnt/k8s-volumes
    server: 192.168.1.100

Persistent Volume Claim manifest:

				
					apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi
  storageClassName: slow

Let’s look at those and try to understand what’s going on:

PV:
We create the nfs-persistent-volume with total storage of 15 GB. For that we use our NFS volume that is available under the IP 192.168.1.100 (you need to change this IP to the one where your drive is mounted). Additionally, we configure the access mode, storage class and few other options for that specific volume – for more information on those check this part of k8s docs.
PVC:
We create the test-pvc claim that requests 10 GB of storage with specific parameters. If the cluster contains a PV with parameters matching those in our requests, the claim will be fulfilled. Otherwise it will be pending until a matching resource can be found.

You should save those two manifests to two separate yml files (e.g. k8s-pv-manifest.yml and k8s-pvc-manifest.yml). Now, to deploy those to your cluster just do:

				
					kubectl apply -f k8s-pv-manifest.yml
kubectl apply -f k8s-pvc-manifest.yml

If everything went fine you should see:

persistentvolume/nfs-persistent-volume created
persistentvolumeclaim/test-pvc created

We can use the Kubernetes Dashboard to check whether all went well. After opening the dashboard, select Persistent Volumes in the menu on the left (under Cluster). You should now see a list of all your volumes. Go ahead and select the volume we just created – you should see the details of that volume which should resemble:

Similarly, when you select Persistent Volume Claims under Config and Storage, you should see our claim and its details:

Voilà! All looks good. We have configured a PV and a PVC bound to it that can now be used by some pods to store data. More details on that coming soon.

In case you want to see what happens when your claim cannot be satisfied, just change the name of the claim in your PVC manifest and alter the storage (under resources: requests) to a higher number, e.g. 20Gi. After applying that manifest you should see in the k8s dashboard that its status is Pending – your PV does not have enough space to bind that claim. Feel free to remove that claim by clicking on the bin icon in the top-right corner.

That’s all for now! We are nearly ready to start using our cluster for some useful computation. Before that, however, we may still try to test all of the components from parts 1-3 so that you can get a full picture of how they play together. Until then!

References

more on the file system table
Kubernetes persistent volumes docs
storage configuration tutorial on RPi by Marco Pivetta (@Ocramius)

Michał

Hey! I’m a protein biochemist by training with a more recent software development experience. Here, I’ll be trying to combine a bit of both – let’s see though where we’ll end up with that 🙂