Using Tigris as a Filesystem

1. nfs-subdir-external-provisioner

   1 50 3,013 2.6 Shell

   Dynamic sub-dir volume provisioner on a remote NFS server.

   

   Kubernetes workloads are stateless by default. This lets you represent a lot of valid services (it's kinda like Heroku by default), but many times you will need persistent storage such as filesystem, relational, or object storage. In my homelab cluster, I have two StorageClasses: NFS mounting folders from my NAS (via the nfs-subdir-external-provisioner), and Longhorn. I ended up going object-storage native for this project so that I didn't have to worry about exhausting the storage on my homelab machines.

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. k8s-csi-s3

   2 4 857 5.0 Go

   GeeseFS-based CSI for mounting S3 buckets as PersistentVolumes

   

   I did some digging and found out about Yandex's csi-s3. It's a StorageClass implementation that uses S3 buckets via geesefs as its backing store instead of storage devices (rotational drives, SSD, EBS, etc.). Unlike a lot of other StorageClass implementations I've tried this year, csi-s3 was really really easy to install. All I had to do was apply the release with helmfile and it was up:

4. rclone

   3 995 57,821 9.8 Go

   "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

   

   Once that was done, I made a bucket (imaginatively named pvfm) and copied the data over with aws s3 sync. I probably could have gotten better performance out of rclone or s5cmd (or if I copied the data to my NAS with its 2.5 gigabit NIC), but I started it, went to sleep, and when I woke up it was done. When I looked back over the logs, I noticed that the main reason why it took so long was that a lot of the older files had many small files alongside of them (.cue sheets listing when each track started and stopped in the DJ set). Tigris handles many small files efficiently, but aws s3 sync didn't properly recycle HTTP connections so uploading a small file was way more costly than it probably should have been. Otherwise I was hitting the limits of the gigabit ethernet card in my shellbox. Sweet!

5. s5cmd

   4 15 4,077 4.4 Go

   Parallel S3 and local filesystem execution tool.

   

   Once that was done, I made a bucket (imaginatively named pvfm) and copied the data over with aws s3 sync. I probably could have gotten better performance out of rclone or s5cmd (or if I copied the data to my NAS with its 2.5 gigabit NIC), but I started it, went to sleep, and when I woke up it was done. When I looked back over the logs, I noticed that the main reason why it took so long was that a lot of the older files had many small files alongside of them (.cue sheets listing when each track started and stopped in the DJ set). Tigris handles many small files efficiently, but aws s3 sync didn't properly recycle HTTP connections so uploading a small file was way more costly than it probably should have been. Otherwise I was hitting the limits of the gigabit ethernet card in my shellbox. Sweet!

6. longhorn

   5 86 7,764 9.4 Shell

   Cloud-Native distributed storage built on and for Kubernetes

   

   Kubernetes workloads are stateless by default. This lets you represent a lot of valid services (it's kinda like Heroku by default), but many times you will need persistent storage such as filesystem, relational, or object storage. In my homelab cluster, I have two StorageClasses: NFS mounting folders from my NAS (via the nfs-subdir-external-provisioner), and Longhorn. I ended up going object-storage native for this project so that I didn't have to worry about exhausting the storage on my homelab machines.

7. x

   6 1 324 9.6 Go

   Various tools, trinkets and experiments (by Xe)

   

   So in order to create a PersistentVolumeClaim pointing at the pvfm bucket, I needed to make both a PersistentVolume and the PersistentVolumeClaim at the same time. This was fairly simple thanks to one of their examples, and I made a single config with both of them in it.

8. cert-manager

   7 128 13,844 9.9 Go

   Automatically provision and manage TLS certificates in Kubernetes

   

   Applying the Deployment, Service, and Ingress went off without a hitch. cert-manager minted a new certificate and External DNS set the DNS target for me. All that was left was to make sure it worked.

Suggest a related project