k3s-ansible VS rook

If you're down for some purchases, I recommend wholeheartedly to get the turing pi https://turingpi.com/ some raspberry modules and k3s-ansible https://github.com/k3s-io/k3s-ansible

I tried k3s recently and installing it was easy using k3s-ansible https://github.com/k3s-io/k3s-ansible

I use some ansible playbooks and when I am ready to update the version I update the variable in the playbooks. https://github.com/k3s-io/k3s-ansible

root@k3s1:~# git clone https://github.com/k3s-io/k3s-ansible Cloning into 'k3s-ansible'... remote: Enumerating objects: 922, done. remote: Total 922 (delta 0), reused 0 (delta 0), pack-reused 922 Receiving objects: 100% (922/922), 116.25 KiB | 3.32 MiB/s, done. Resolving deltas: 100% (351/351), done. root@k3s1:~# cd k3s-ansible/ root@k3s1:~/k3s-ansible# cp -R inventory/sample inventory/my-cluster root@k3s1:~/k3s-ansible# cat inventory/my-cluster/group_vars/all.yml --- k3s_version: v1.24.7+k3s1 ansible_user: ubuntu systemd_dir: /etc/systemd/system master_ip: "{{ hostvars[groups['master'][0]]['ansible_host'] | default(groups['master'][0]) }}" extra_server_args: "" extra_agent_args: "" root@k3s1:~/k3s-ansible# cat inventory/my-cluster/hosts.ini [master] 10.42.2.6 [node] 10.42.0.7 10.42.1.9 [k3s_cluster:children] master node root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# apt install python3-pip -y && pip install ansible root@k3s1:~/k3s-ansible# ansible-playbook site.yml -i inventory/my-cluster/hosts.ini PLAY [k3s_cluster] **************************************************************************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:03 +0000 (0:00:00.013) 0:00:00.013 ***** ok: [10.42.2.6] ok: [10.42.1.9] ok: [10.42.0.7] TASK [prereq : Set SELinux to disabled state] ************************************************************************************************************************************* Thursday 10 November 2022 23:20:04 +0000 (0:00:01.535) 0:00:01.548 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Enable IPv4 forwarding] ******************************************************************************************************************************************** Thursday 10 November 2022 23:20:04 +0000 (0:00:00.058) 0:00:01.607 ***** changed: [10.42.2.6] changed: [10.42.1.9] changed: [10.42.0.7] TASK [prereq : Enable IPv6 forwarding] ******************************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.318) 0:00:01.925 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Add br_netfilter to /etc/modules-load.d/] ************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.053) 0:00:01.978 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Load br_netfilter] ************************************************************************************************************************************************* Thursday 10 November 2022 23:20:05 +0000 (0:00:00.051) 0:00:02.030 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Set bridge-nf-call-iptables (just to be sure)] ********************************************************************************************************************* Thursday 10 November 2022 23:20:05 +0000 (0:00:00.052) 0:00:02.082 ***** skipping: [10.42.2.6] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.2.6] => (item=net.bridge.bridge-nf-call-ip6tables) skipping: [10.42.0.7] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.0.7] => (item=net.bridge.bridge-nf-call-ip6tables) skipping: [10.42.1.9] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.1.9] => (item=net.bridge.bridge-nf-call-ip6tables) TASK [prereq : Add /usr/local/bin to sudo secure_path] **************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.072) 0:00:02.154 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [download : Download k3s binary x64] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.103) 0:00:02.258 ***** [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually changed: [10.42.2.6] changed: [10.42.0.7] changed: [10.42.1.9] TASK [download : Download k3s binary arm64] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:10.036) 0:00:12.295 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [download : Download k3s binary armhf] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:00.067) 0:00:12.362 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Test for raspberry pi /proc/cpuinfo] ************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:00.084) 0:00:12.447 ***** ok: [10.42.2.6] ok: [10.42.0.7] ok: [10.42.1.9] TASK [raspberrypi : Test for raspberry pi /proc/device-tree/model] **************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.359) 0:00:12.806 ***** ok: [10.42.2.6] ok: [10.42.0.7] ok: [10.42.1.9] TASK [raspberrypi : Set raspberry_pi fact to true] ******************************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.255) 0:00:13.062 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution to Raspbian] ************************************************************************************************************************ Thursday 10 November 2022 23:20:16 +0000 (0:00:00.060) 0:00:13.123 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution to Raspbian (ARM64 on Debian Buster)] *********************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.112) 0:00:13.235 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution_major_version] ********************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.073) 0:00:13.309 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : execute OS related tasks on the Raspberry Pi] ***************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.080) 0:00:13.389 ***** skipping: [10.42.2.6] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) skipping: [10.42.0.7] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) skipping: [10.42.1.9] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) PLAY [master] ********************************************************************************************************************************************************************* TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:16 +0000 (0:00:00.125) 0:00:13.515 ***** ok: [10.42.2.6] TASK [k3s/master : Copy K3s service file] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:17 +0000 (0:00:00.660) 0:00:14.175 ***** changed: [10.42.2.6] TASK [k3s/master : Enable and check K3s service] ********************************************************************************************************************************** Thursday 10 November 2022 23:20:18 +0000 (0:00:00.688) 0:00:14.864 ***** changed: [10.42.2.6] TASK [k3s/master : Wait for node-token] ******************************************************************************************************************************************* Thursday 10 November 2022 23:20:29 +0000 (0:00:11.160) 0:00:26.025 ***** ok: [10.42.2.6] TASK [k3s/master : Register node-token file access mode] ************************************************************************************************************************** Thursday 10 November 2022 23:20:29 +0000 (0:00:00.458) 0:00:26.483 ***** ok: [10.42.2.6] TASK [k3s/master : Change file access node-token] ********************************************************************************************************************************* Thursday 10 November 2022 23:20:30 +0000 (0:00:00.329) 0:00:26.812 ***** changed: [10.42.2.6] TASK [k3s/master : Read node-token from master] *********************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.422) 0:00:27.234 ***** ok: [10.42.2.6] TASK [k3s/master : Store Master node-token] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.303) 0:00:27.537 ***** ok: [10.42.2.6] TASK [k3s/master : Restore node-token file access] ******************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.041) 0:00:27.579 ***** changed: [10.42.2.6] TASK [k3s/master : Create directory .kube] **************************************************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.232) 0:00:27.812 ***** changed: [10.42.2.6] TASK [k3s/master : Copy config file to user home directory] *********************************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.310) 0:00:28.122 ***** changed: [10.42.2.6] TASK [k3s/master : Replace https://localhost:6443 by https://master-ip:6443] ****************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.211) 0:00:28.334 ***** changed: [10.42.2.6] TASK [k3s/master : Create kubectl symlink] **************************************************************************************************************************************** Thursday 10 November 2022 23:20:32 +0000 (0:00:00.693) 0:00:29.027 ***** changed: [10.42.2.6] TASK [k3s/master : Create crictl symlink] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:32 +0000 (0:00:00.187) 0:00:29.215 ***** changed: [10.42.2.6] PLAY [node] *********************************************************************************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:32 +0000 (0:00:00.209) 0:00:29.424 ***** ok: [10.42.1.9] ok: [10.42.0.7] TASK [k3s/node : Copy K3s service file] ******************************************************************************************************************************************* Thursday 10 November 2022 23:20:33 +0000 (0:00:00.724) 0:00:30.149 ***** changed: [10.42.1.9] changed: [10.42.0.7] TASK [k3s/node : Enable and check K3s service] ************************************************************************************************************************************ Thursday 10 November 2022 23:20:34 +0000 (0:00:00.538) 0:00:30.687 ***** changed: [10.42.1.9] changed: [10.42.0.7] PLAY RECAP ************************************************************************************************************************************************************************ 10.42.0.7 : ok=8 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 10.42.1.9 : ok=8 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 10.42.2.6 : ok=19 changed=11 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 Thursday 10 November 2022 23:20:44 +0000 (0:00:10.395) 0:00:41.082 ***** =============================================================================== k3s/master : Enable and check K3s service --------------------------------------------------------------------------------------------------------------------------------- 11.16s k3s/node : Enable and check K3s service ----------------------------------------------------------------------------------------------------------------------------------- 10.40s download : Download k3s binary x64 ---------------------------------------------------------------------------------------------------------------------------------------- 10.04s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.54s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.72s k3s/master : Replace https://localhost:6443 by https://master-ip:6443 ------------------------------------------------------------------------------------------------------ 0.69s k3s/master : Copy K3s service file ----------------------------------------------------------------------------------------------------------------------------------------- 0.69s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.66s k3s/node : Copy K3s service file ------------------------------------------------------------------------------------------------------------------------------------------- 0.54s k3s/master : Wait for node-token ------------------------------------------------------------------------------------------------------------------------------------------- 0.46s k3s/master : Change file access node-token --------------------------------------------------------------------------------------------------------------------------------- 0.42s raspberrypi : Test for raspberry pi /proc/cpuinfo -------------------------------------------------------------------------------------------------------------------------- 0.36s k3s/master : Register node-token file access mode -------------------------------------------------------------------------------------------------------------------------- 0.33s prereq : Enable IPv4 forwarding -------------------------------------------------------------------------------------------------------------------------------------------- 0.32s k3s/master : Create directory .kube ---------------------------------------------------------------------------------------------------------------------------------------- 0.31s k3s/master : Read node-token from master ----------------------------------------------------------------------------------------------------------------------------------- 0.30s raspberrypi : Test for raspberry pi /proc/device-tree/model ---------------------------------------------------------------------------------------------------------------- 0.26s k3s/master : Restore node-token file access -------------------------------------------------------------------------------------------------------------------------------- 0.23s k3s/master : Copy config file to user home directory ----------------------------------------------------------------------------------------------------------------------- 0.21s k3s/master : Create crictl symlink ----------------------------------------------------------------------------------------------------------------------------------------- 0.21s

Plus there is a pretty good ansible playbook https://github.com/k3s-io/k3s-ansible

We have develop ansible role to do that. We did not put it as opensource yet. But there is one in the comunity which looks good: https://github.com/k3s-io/k3s-ansible

For the Ansible part, have a look at techno-tim/k3s-ansible for a basic k3s (not the full thing) with MetalLB ansible setup. There's also the original one (based on Traefik): k3s-io/k3s-ansible. Modfiy hosts and vars accordingly to your needs. Run your terraform plan, then run the ansible playbook once you've modified the vars accordingly. They're good starters, but for a full blown k8s ansible recipe, I'll be able to help you when I come back from holidays :) (16th of August).

I have some experience with Ceph, both for work, and with homelab-y stuff.

First, bear in mind that Ceph is a distributed storage system - so the idea is that you will have multiple nodes.

For learning, you can definitely virtualise it all on a single box - but you'll have a better time with discrete physical machines.

Also, Ceph does prefer physical access to disks (similar to ZFS).

And you do need decent networking connectivity - I think that's the main thing people think of, when they think of high hardware requirements for Ceph. Ideally 10Gbe at the minimum - although more if you want higher performance - there can be a lot of network traffic, particularly with things like backfill. (25Gbps if you can find that gear cheap for homelab - 50Gbps is a technological dead-end. 100Gbps works well).

But honestly, for a homelab, a cheap mini PC or NUC with 10Gbe will work fine, and you should get acceptable performance, and it'll be good for learning.

You can install Ceph directly on bare-metal, or if you want to do the homelab k8s route, you can use Rook (https://rook.io/).

Hope this helps, and good luck! Let me know if you have any other questions.

Another option is to leverage a Kubernetes-native distributed storage solution such as Rook Ceph as the storage backend for stateful components running on Kubernetes. This has the benefit of simplifying application configuration while addressing business requirements for data backup and recovery such as the ability to take volume snapshots at a regular interval and perform application-level data recovery in case of a disaster.

This is beyond your question but might help someone else: I switch from docker-compose to kubernetes for my home lab a while ago. The storage solution I've settled on is Rook. It was a bit of up-front work learning how to get it up but now that it's done my storage is automatically managed by Ceph. I can swap out drives and Ceph basically takes care of everything itself.

The stumbling point I am at is I want to use rook.io(Ceph) as my storage solution for the cluster. The Ceph prerequisites are one of the following:

Have you looked at Rook?

I've briefly run ceph in an external mode, you can actually use a rook deployment to manage it (sort of). Here is the documentation for doing that. For me it didn't pass my testing phase because I need better networking equipment before I can try that.

This article explains the data corruption issue happened in Rook in 2021. The root cause lies in an unexpected place and can also occurs in all Ceph environment. It's interesting that Rook had started to encounter this problem recently even though this problem has existed for a long time. It's due to a series of coincidences. I wrote this article because the word "Atari" used in a non-historical context in 2021.

Rook (this is a nice article for Rook NFS)

Storage: Favor any distributed storage you know to start with for Persistent Volumes: Ceph maybe via rook.io, Longhorn if you go rancher etc

I've dealt with a lot of issues that are very close to just unplugging a node. Unfortunately on node lost, my stateful workloads using rook-ceph block storage won't migrate over to another node automatically due to an issue with rook. Stateless apps (ingress nginx, etc..) not using rook-ceph block failover to another node just fine. I've kind of accepted this for now and I know Longhorn has a feature that makes this work but I find rook-ceph to be more stable for my workloads.