longhorn VS k3sup

For my primary CSI provider, I use Longhorn, and to administrate my clusters, I use the Kubernetes Dashboard. In front of all of my private dashboards, I use Traefik Forward Auth to limit who can access them.

Maybe with https://longhorn.io

For data must be protected, I use Longhorn https://longhorn.io/ which is basically iSCSI/NFS with replication.

Backups are definitely necessary. The problem though is that you cant detect by default if you are backing up corrupted data. By now i found three approaches: 1. Use a filesystem that detects and fixes errors (e. g. btrfs or zfs) 2. Use par2 (parchive) to generate checksums on your files manually 3. I found the following issue on the k8s storage solution I currently use: https://github.com/longhorn/longhorn/issues/3198 For now I am waiting for option 3 getting merged. In the long run I will upgrade my laptop to btrfs and for now use par2. Hope that provides a starting point. Feel free to add your ideas!

You can use the Longhorn iSCSI installer script as an example. https://github.com/longhorn/longhorn/blob/master/deploy/prerequisite/longhorn-iscsi-installation.yaml

Rancher's Longhorn (even though it's only a single-node)

Application storage now: yes, NAS is probably the best way to go about it. I believe k3s uses some kind of "local storage" by default but there's plenty of alternative Container Storage Interfaces (CSI) out there. Longhorn worked well for me and it covers most uses cases I care about: snapshots, backups, and replication. I don't know about any "NAS providers" though, and once again there will be performance drawbacks with having your storage that externalized.

For me longhorn works really well as my storage backend.

Longhorn is a lightweight, reliable, and powerful distributed block storage system for Kubernetes.

You mentioned storage... That's always tricky but provisioning persistent volumes with https://longhorn.io is really simple... It's also easy to setup, the helm chart is wonderful

$ curl -sLS https://get.k3sup.dev | sh x86_64 Downloading package https://github.com/alexellis/k3sup/releases/download/0.12.12/k3sup as /home/ec2-user/k3sup Download complete. ============================================================ The script was run as a user who is unable to write to /usr/local/bin. To complete the installation the following commands may need to be run manually. ============================================================ sudo cp k3sup /usr/local/bin/k3sup ================================================================ alexellis's work on k3sup needs your support https://github.com/sponsors/alexellis ================================================================ No nos devolverá nada, pero podremos correr lo siguiente para saber si k3sup efectivamente se instalo:

I am used to k3sup https://github.com/alexellis/k3sup, this project seems to provide more tuning configs.

[email protected]:~# curl -sLS https://get.k3sup.dev | sh x86_64 Downloading package https://github.com/alexellis/k3sup/releases/download/0.12.12/k3sup as /tmp/k3sup Download complete. Running with sufficient permissions to attempt to move k3sup to /usr/local/bin New version of k3sup installed to /usr/local/bin _ _____ | | _| ___/___ _ _ _ __ | |/ / |_ \/ __| | | | '_ \ | < ___) \__ \ |_| | |_) | |_|\_\ ____/|___ /\ __,_| .__ / |_| bootstrap K3s over SSH in < 60s 🚀 🐳 k3sup needs your support: https://github.com/sponsors/alexellis Version: 0.12.12 Git Commit: 02c7a775b9914b9dcf3b90fa7935eb347b7979e7 ================================================================ alexellis's work on k3sup needs your support https://github.com/sponsors/alexellis ================================================================ [email protected]:~# k3sup --help Usage: k3sup [flags] k3sup [command] Examples: # Install k3s on a server with embedded etcd k3sup install \ --cluster \ --host $SERVER_1 \ --user $SERVER_1_USER \ --k3s-channel stable # Join a second server k3sup join \ --server \ --host $SERVER_2 \ --user $SERVER_2_USER \ --server-host $SERVER_1 \ --server-user $SERVER_1_USER \ --k3s-channel stable # Join an agent to the cluster k3sup join \ --host $SERVER_1 \ --user $SERVER_1_USER \ --k3s-channel stable 🐳 k3sup needs your support: https://github.com/sponsors/alexellis Available Commands: completion Generate the autocompletion script for the specified shell help Help about any command install Install k3s on a server via SSH join Install the k3s agent on a remote host and join it to an existing server ready Check if a cluster is ready using kubectl. update Print update instructions version Print the version Flags: -h, --help help for k3sup Use "k3sup [command] --help" for more information about a command. [email protected]:~# export IP=10.137.67.183 [email protected]:~# k3sup install --ip $IP --user ubuntu Running: k3sup install 2022/12/25 14:30:07 10.137.67.183 Public IP: 10.137.67.183 [INFO] Finding release for channel stable [INFO] Using v1.25.4+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s.service [INFO] systemd: Enabling k3s unit Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service. [INFO] systemd: Starting k3s Result: [INFO] Finding release for channel stable [INFO] Using v1.25.4+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s.service [INFO] systemd: Enabling k3s unit [INFO] systemd: Starting k3s Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service. Saving file to: /root/kubeconfig # Test your cluster with: export KUBECONFIG=/root/kubeconfig kubectl config use-context default kubectl get node -o wide 🐳 k3sup needs your support: https://github.com/sponsors/alexellis [email protected]:~# export AGENT1_IP=10.137.67.177 [email protected]:~# k3sup join --ip $AGENT1_IP --server-ip $IP --user ubuntu Running: k3sup join Server IP: 10.137.67.183 K107122269d6d9c54733f9c835439e668a98b6b0a9c5b9cc7ce7272b0cc9b294b09::server:ccc6e02aac3498cabc1b8ab7b3f334c1 [INFO] Finding release for channel stable [INFO] Using v1.25.4+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service [INFO] systemd: Enabling k3s-agent unit Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service. [INFO] systemd: Starting k3s-agent Logs: Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service. Output: [INFO] Finding release for channel stable [INFO] Using v1.25.4+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service [INFO] systemd: Enabling k3s-agent unit [INFO] systemd: Starting k3s-agent 🐳 k3sup needs your support: https://github.com/sponsors/alexellis [email protected]:~# export AGENT2_IP=10.137.67.222 [email protected]:~# k3sup join --ip $AGENT2_IP --server-ip $IP --user ubuntu Running: k3sup join Server IP: 10.137.67.183 K107122269d6d9c54733f9c835439e668a98b6b0a9c5b9cc7ce7272b0cc9b294b09::server:ccc6e02aac3498cabc1b8ab7b3f334c1 [INFO] Finding release for channel stable [INFO] Using v1.25.4+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service [INFO] systemd: Enabling k3s-agent unit Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service. [INFO] systemd: Starting k3s-agent Logs: Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service. Output: [INFO] Finding release for channel stable [INFO] Using v1.25.4+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service [INFO] systemd: Enabling k3s-agent unit [INFO] systemd: Starting k3s-agent 🐳 k3sup needs your support: https://github.com/sponsors/alexellis

There is a convenience utility for K3S called k3sup.

From my laptop, not one of the Raspberry Pi nodes, run k3sup, it's super easy. If you haven't figured it out by now, I'm a big fan of k3sup.

Best way to install it is ketchup: https://github.com/alexellis/k3sup

Personal preference, Terraform the relevant Ubuntu VM's (spec according to requirements) and then use K3 sup (https://github.com/alexellis/k3sup) and remove klipper and traefik using the relevant switches

it works fine for me. I use k3sup https://github.com/alexellis/k3sup and some terraform provisioning scripts.

I have enjoyed using k3s and k3sup for running my at home cluster.

I use https://k3s.io and it's super simple. You will have a basic cluster running in less than 5 minutes. There's even an unofficial installer for it (https://github.com/alexellis/k3sup) which makes the bootstrapping even easier.