k3s-ansible VS talos

If you're down for some purchases, I recommend wholeheartedly to get the turing pi https://turingpi.com/ some raspberry modules and k3s-ansible https://github.com/k3s-io/k3s-ansible

I tried k3s recently and installing it was easy using k3s-ansible https://github.com/k3s-io/k3s-ansible

I use some ansible playbooks and when I am ready to update the version I update the variable in the playbooks. https://github.com/k3s-io/k3s-ansible

root@k3s1:~# git clone https://github.com/k3s-io/k3s-ansible Cloning into 'k3s-ansible'... remote: Enumerating objects: 922, done. remote: Total 922 (delta 0), reused 0 (delta 0), pack-reused 922 Receiving objects: 100% (922/922), 116.25 KiB | 3.32 MiB/s, done. Resolving deltas: 100% (351/351), done. root@k3s1:~# cd k3s-ansible/ root@k3s1:~/k3s-ansible# cp -R inventory/sample inventory/my-cluster root@k3s1:~/k3s-ansible# cat inventory/my-cluster/group_vars/all.yml --- k3s_version: v1.24.7+k3s1 ansible_user: ubuntu systemd_dir: /etc/systemd/system master_ip: "{{ hostvars[groups['master'][0]]['ansible_host'] | default(groups['master'][0]) }}" extra_server_args: "" extra_agent_args: "" root@k3s1:~/k3s-ansible# cat inventory/my-cluster/hosts.ini [master] 10.42.2.6 [node] 10.42.0.7 10.42.1.9 [k3s_cluster:children] master node root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# apt install python3-pip -y && pip install ansible root@k3s1:~/k3s-ansible# ansible-playbook site.yml -i inventory/my-cluster/hosts.ini PLAY [k3s_cluster] **************************************************************************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:03 +0000 (0:00:00.013) 0:00:00.013 ***** ok: [10.42.2.6] ok: [10.42.1.9] ok: [10.42.0.7] TASK [prereq : Set SELinux to disabled state] ************************************************************************************************************************************* Thursday 10 November 2022 23:20:04 +0000 (0:00:01.535) 0:00:01.548 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Enable IPv4 forwarding] ******************************************************************************************************************************************** Thursday 10 November 2022 23:20:04 +0000 (0:00:00.058) 0:00:01.607 ***** changed: [10.42.2.6] changed: [10.42.1.9] changed: [10.42.0.7] TASK [prereq : Enable IPv6 forwarding] ******************************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.318) 0:00:01.925 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Add br_netfilter to /etc/modules-load.d/] ************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.053) 0:00:01.978 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Load br_netfilter] ************************************************************************************************************************************************* Thursday 10 November 2022 23:20:05 +0000 (0:00:00.051) 0:00:02.030 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Set bridge-nf-call-iptables (just to be sure)] ********************************************************************************************************************* Thursday 10 November 2022 23:20:05 +0000 (0:00:00.052) 0:00:02.082 ***** skipping: [10.42.2.6] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.2.6] => (item=net.bridge.bridge-nf-call-ip6tables) skipping: [10.42.0.7] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.0.7] => (item=net.bridge.bridge-nf-call-ip6tables) skipping: [10.42.1.9] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.1.9] => (item=net.bridge.bridge-nf-call-ip6tables) TASK [prereq : Add /usr/local/bin to sudo secure_path] **************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.072) 0:00:02.154 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [download : Download k3s binary x64] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.103) 0:00:02.258 ***** [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually changed: [10.42.2.6] changed: [10.42.0.7] changed: [10.42.1.9] TASK [download : Download k3s binary arm64] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:10.036) 0:00:12.295 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [download : Download k3s binary armhf] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:00.067) 0:00:12.362 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Test for raspberry pi /proc/cpuinfo] ************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:00.084) 0:00:12.447 ***** ok: [10.42.2.6] ok: [10.42.0.7] ok: [10.42.1.9] TASK [raspberrypi : Test for raspberry pi /proc/device-tree/model] **************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.359) 0:00:12.806 ***** ok: [10.42.2.6] ok: [10.42.0.7] ok: [10.42.1.9] TASK [raspberrypi : Set raspberry_pi fact to true] ******************************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.255) 0:00:13.062 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution to Raspbian] ************************************************************************************************************************ Thursday 10 November 2022 23:20:16 +0000 (0:00:00.060) 0:00:13.123 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution to Raspbian (ARM64 on Debian Buster)] *********************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.112) 0:00:13.235 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution_major_version] ********************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.073) 0:00:13.309 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : execute OS related tasks on the Raspberry Pi] ***************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.080) 0:00:13.389 ***** skipping: [10.42.2.6] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) skipping: [10.42.0.7] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) skipping: [10.42.1.9] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) PLAY [master] ********************************************************************************************************************************************************************* TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:16 +0000 (0:00:00.125) 0:00:13.515 ***** ok: [10.42.2.6] TASK [k3s/master : Copy K3s service file] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:17 +0000 (0:00:00.660) 0:00:14.175 ***** changed: [10.42.2.6] TASK [k3s/master : Enable and check K3s service] ********************************************************************************************************************************** Thursday 10 November 2022 23:20:18 +0000 (0:00:00.688) 0:00:14.864 ***** changed: [10.42.2.6] TASK [k3s/master : Wait for node-token] ******************************************************************************************************************************************* Thursday 10 November 2022 23:20:29 +0000 (0:00:11.160) 0:00:26.025 ***** ok: [10.42.2.6] TASK [k3s/master : Register node-token file access mode] ************************************************************************************************************************** Thursday 10 November 2022 23:20:29 +0000 (0:00:00.458) 0:00:26.483 ***** ok: [10.42.2.6] TASK [k3s/master : Change file access node-token] ********************************************************************************************************************************* Thursday 10 November 2022 23:20:30 +0000 (0:00:00.329) 0:00:26.812 ***** changed: [10.42.2.6] TASK [k3s/master : Read node-token from master] *********************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.422) 0:00:27.234 ***** ok: [10.42.2.6] TASK [k3s/master : Store Master node-token] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.303) 0:00:27.537 ***** ok: [10.42.2.6] TASK [k3s/master : Restore node-token file access] ******************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.041) 0:00:27.579 ***** changed: [10.42.2.6] TASK [k3s/master : Create directory .kube] **************************************************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.232) 0:00:27.812 ***** changed: [10.42.2.6] TASK [k3s/master : Copy config file to user home directory] *********************************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.310) 0:00:28.122 ***** changed: [10.42.2.6] TASK [k3s/master : Replace https://localhost:6443 by https://master-ip:6443] ****************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.211) 0:00:28.334 ***** changed: [10.42.2.6] TASK [k3s/master : Create kubectl symlink] **************************************************************************************************************************************** Thursday 10 November 2022 23:20:32 +0000 (0:00:00.693) 0:00:29.027 ***** changed: [10.42.2.6] TASK [k3s/master : Create crictl symlink] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:32 +0000 (0:00:00.187) 0:00:29.215 ***** changed: [10.42.2.6] PLAY [node] *********************************************************************************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:32 +0000 (0:00:00.209) 0:00:29.424 ***** ok: [10.42.1.9] ok: [10.42.0.7] TASK [k3s/node : Copy K3s service file] ******************************************************************************************************************************************* Thursday 10 November 2022 23:20:33 +0000 (0:00:00.724) 0:00:30.149 ***** changed: [10.42.1.9] changed: [10.42.0.7] TASK [k3s/node : Enable and check K3s service] ************************************************************************************************************************************ Thursday 10 November 2022 23:20:34 +0000 (0:00:00.538) 0:00:30.687 ***** changed: [10.42.1.9] changed: [10.42.0.7] PLAY RECAP ************************************************************************************************************************************************************************ 10.42.0.7 : ok=8 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 10.42.1.9 : ok=8 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 10.42.2.6 : ok=19 changed=11 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 Thursday 10 November 2022 23:20:44 +0000 (0:00:10.395) 0:00:41.082 ***** =============================================================================== k3s/master : Enable and check K3s service --------------------------------------------------------------------------------------------------------------------------------- 11.16s k3s/node : Enable and check K3s service ----------------------------------------------------------------------------------------------------------------------------------- 10.40s download : Download k3s binary x64 ---------------------------------------------------------------------------------------------------------------------------------------- 10.04s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.54s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.72s k3s/master : Replace https://localhost:6443 by https://master-ip:6443 ------------------------------------------------------------------------------------------------------ 0.69s k3s/master : Copy K3s service file ----------------------------------------------------------------------------------------------------------------------------------------- 0.69s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.66s k3s/node : Copy K3s service file ------------------------------------------------------------------------------------------------------------------------------------------- 0.54s k3s/master : Wait for node-token ------------------------------------------------------------------------------------------------------------------------------------------- 0.46s k3s/master : Change file access node-token --------------------------------------------------------------------------------------------------------------------------------- 0.42s raspberrypi : Test for raspberry pi /proc/cpuinfo -------------------------------------------------------------------------------------------------------------------------- 0.36s k3s/master : Register node-token file access mode -------------------------------------------------------------------------------------------------------------------------- 0.33s prereq : Enable IPv4 forwarding -------------------------------------------------------------------------------------------------------------------------------------------- 0.32s k3s/master : Create directory .kube ---------------------------------------------------------------------------------------------------------------------------------------- 0.31s k3s/master : Read node-token from master ----------------------------------------------------------------------------------------------------------------------------------- 0.30s raspberrypi : Test for raspberry pi /proc/device-tree/model ---------------------------------------------------------------------------------------------------------------- 0.26s k3s/master : Restore node-token file access -------------------------------------------------------------------------------------------------------------------------------- 0.23s k3s/master : Copy config file to user home directory ----------------------------------------------------------------------------------------------------------------------- 0.21s k3s/master : Create crictl symlink ----------------------------------------------------------------------------------------------------------------------------------------- 0.21s

Plus there is a pretty good ansible playbook https://github.com/k3s-io/k3s-ansible

We have develop ansible role to do that. We did not put it as opensource yet. But there is one in the comunity which looks good: https://github.com/k3s-io/k3s-ansible

For the Ansible part, have a look at techno-tim/k3s-ansible for a basic k3s (not the full thing) with MetalLB ansible setup. There's also the original one (based on Traefik): k3s-io/k3s-ansible. Modfiy hosts and vars accordingly to your needs. Run your terraform plan, then run the ansible playbook once you've modified the vars accordingly. They're good starters, but for a full blown k8s ansible recipe, I'll be able to help you when I come back from holidays :) (16th of August).

Super cool. I always enjoy reading about systems that challenge, well, "ossified" assumptions. An OS not providing a shell, for example? Madness! ... or is it genius, if the OS has a specific purpose...? It's thought-provoking, if nothing else.

I'm a bit skeptical of parts. For instance, the "init" binary being less than 400 lines of golang - wow! And sure, main.go [1] is less than 400 lines and very readable. Then you squint at the list of imported packages, or look to the left at the directory list and realize main.go isn't nearly the entire init binary.

That `talosctl list` invocation [2] didn't escape my notice either. Sure, the base OS may have only a handful of binaries - how many of those traditional utilities have been stuffed into the API server? Not that I disagree with the approach! I think every company eventually replaces direct shell access with a daemon like this. It's just that "binary footprint" can get a bit funny if you have a really sophisticated API server sitting somewhere.

[1]: https://github.com/siderolabs/talos/blob/main/internal/app/m...

[2]: https://www.talos.dev/v1.6/reference/cli/#talosctl-list

Where `kube.cue` sets reasonable defaults (e.g. image is /). The "cluster" runs on a mini PC in my basement, and I have a small Digital Ocean VM with a static IP acting as an ingress (networking via Tailscale). Backups to cloud storage with restic, alerting/monitoring with Prometheus/Grafana, Caddy/Tailscale for local ingress.

[1] https://www.talos.dev/

[2] https://cuelang.org/

Looks somewhat similar to the talos Linux project[1]

[1] https://www.talos.dev/

Talos Linux basically implements their entire userspace in Go and its similar to BottleRocketOS, because it is designed to host Kubernetes.

https://www.talos.dev/

You might be surprised to find that Talos os (linux distro for kubernetes) mostly uses Go: https://github.com/siderolabs/talos

I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use.

The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environment setups.

I even use it with Longhorn https://longhorn.io/ for shared block storage on the mini cluster.

If anyone uses it with MicroOS, just make sure you switch to kured https://kured.dev/ for the transactional-updates reboot method.

I'd love to compare it against Talos https://www.talos.dev/ but their lack of support for a persistent storage partition (only separate storage device) really hurts most small home / office usage I'd want to try.

If you’re interested in something not AWS check out Talos https://www.talos.dev/

It’s been around longer than Bottlerocket

Can't talk about work, but my homelab is Azure and Oracle managed k8s (AKS/OKE), with onprem Talos soon (Turing Pi 2). My Flux monorepo has the details. OKE performs noticably worse (update cycle, features, control plane performance), but it provides 4 ARM cores and 24GB RAM free so I can't complain

Talos