ansible-role-k3s VS k3s-ansible

There is another very good ansible role for k3s, actually better than mine

if you have experience with ansible you can use https://github.com/PyratLabs/ansible-role-k3s

Not OP but there's an excellent galaxy ansible role for k3s and it's idempotent, so if you want to upgrade to a newer version of k3s or tweak settings, or add a node it's supported.

Take it to the next level and let git be the source of truth for k8s using Flux! My home cluster repo is exactly that plus I'm using the ansible-k3s-role Galaxy role. It's based on this k8s at home template.

If you are comfortable with ansible there's a nice Galaxy role that I use.

Experience has been good so far but this is the first I saw of KubeEdge. I’m going to give it an evaluation. I used Ansible https://github.com/PyratLabs/ansible-role-k3s to set up the cluster with some minor adaptions and cleanup scripts.

If you're down for some purchases, I recommend wholeheartedly to get the turing pi https://turingpi.com/ some raspberry modules and k3s-ansible https://github.com/k3s-io/k3s-ansible

I tried k3s recently and installing it was easy using k3s-ansible https://github.com/k3s-io/k3s-ansible

I use some ansible playbooks and when I am ready to update the version I update the variable in the playbooks. https://github.com/k3s-io/k3s-ansible

root@k3s1:~# git clone https://github.com/k3s-io/k3s-ansible Cloning into 'k3s-ansible'... remote: Enumerating objects: 922, done. remote: Total 922 (delta 0), reused 0 (delta 0), pack-reused 922 Receiving objects: 100% (922/922), 116.25 KiB | 3.32 MiB/s, done. Resolving deltas: 100% (351/351), done. root@k3s1:~# cd k3s-ansible/ root@k3s1:~/k3s-ansible# cp -R inventory/sample inventory/my-cluster root@k3s1:~/k3s-ansible# cat inventory/my-cluster/group_vars/all.yml --- k3s_version: v1.24.7+k3s1 ansible_user: ubuntu systemd_dir: /etc/systemd/system master_ip: "{{ hostvars[groups['master'][0]]['ansible_host'] | default(groups['master'][0]) }}" extra_server_args: "" extra_agent_args: "" root@k3s1:~/k3s-ansible# cat inventory/my-cluster/hosts.ini [master] 10.42.2.6 [node] 10.42.0.7 10.42.1.9 [k3s_cluster:children] master node root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# apt install python3-pip -y && pip install ansible root@k3s1:~/k3s-ansible# ansible-playbook site.yml -i inventory/my-cluster/hosts.ini PLAY [k3s_cluster] **************************************************************************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:03 +0000 (0:00:00.013) 0:00:00.013 ***** ok: [10.42.2.6] ok: [10.42.1.9] ok: [10.42.0.7] TASK [prereq : Set SELinux to disabled state] ************************************************************************************************************************************* Thursday 10 November 2022 23:20:04 +0000 (0:00:01.535) 0:00:01.548 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Enable IPv4 forwarding] ******************************************************************************************************************************************** Thursday 10 November 2022 23:20:04 +0000 (0:00:00.058) 0:00:01.607 ***** changed: [10.42.2.6] changed: [10.42.1.9] changed: [10.42.0.7] TASK [prereq : Enable IPv6 forwarding] ******************************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.318) 0:00:01.925 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Add br_netfilter to /etc/modules-load.d/] ************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.053) 0:00:01.978 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Load br_netfilter] ************************************************************************************************************************************************* Thursday 10 November 2022 23:20:05 +0000 (0:00:00.051) 0:00:02.030 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Set bridge-nf-call-iptables (just to be sure)] ********************************************************************************************************************* Thursday 10 November 2022 23:20:05 +0000 (0:00:00.052) 0:00:02.082 ***** skipping: [10.42.2.6] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.2.6] => (item=net.bridge.bridge-nf-call-ip6tables) skipping: [10.42.0.7] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.0.7] => (item=net.bridge.bridge-nf-call-ip6tables) skipping: [10.42.1.9] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.1.9] => (item=net.bridge.bridge-nf-call-ip6tables) TASK [prereq : Add /usr/local/bin to sudo secure_path] **************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.072) 0:00:02.154 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [download : Download k3s binary x64] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.103) 0:00:02.258 ***** [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually changed: [10.42.2.6] changed: [10.42.0.7] changed: [10.42.1.9] TASK [download : Download k3s binary arm64] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:10.036) 0:00:12.295 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [download : Download k3s binary armhf] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:00.067) 0:00:12.362 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Test for raspberry pi /proc/cpuinfo] ************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:00.084) 0:00:12.447 ***** ok: [10.42.2.6] ok: [10.42.0.7] ok: [10.42.1.9] TASK [raspberrypi : Test for raspberry pi /proc/device-tree/model] **************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.359) 0:00:12.806 ***** ok: [10.42.2.6] ok: [10.42.0.7] ok: [10.42.1.9] TASK [raspberrypi : Set raspberry_pi fact to true] ******************************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.255) 0:00:13.062 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution to Raspbian] ************************************************************************************************************************ Thursday 10 November 2022 23:20:16 +0000 (0:00:00.060) 0:00:13.123 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution to Raspbian (ARM64 on Debian Buster)] *********************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.112) 0:00:13.235 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution_major_version] ********************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.073) 0:00:13.309 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : execute OS related tasks on the Raspberry Pi] ***************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.080) 0:00:13.389 ***** skipping: [10.42.2.6] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) skipping: [10.42.0.7] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) skipping: [10.42.1.9] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) PLAY [master] ********************************************************************************************************************************************************************* TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:16 +0000 (0:00:00.125) 0:00:13.515 ***** ok: [10.42.2.6] TASK [k3s/master : Copy K3s service file] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:17 +0000 (0:00:00.660) 0:00:14.175 ***** changed: [10.42.2.6] TASK [k3s/master : Enable and check K3s service] ********************************************************************************************************************************** Thursday 10 November 2022 23:20:18 +0000 (0:00:00.688) 0:00:14.864 ***** changed: [10.42.2.6] TASK [k3s/master : Wait for node-token] ******************************************************************************************************************************************* Thursday 10 November 2022 23:20:29 +0000 (0:00:11.160) 0:00:26.025 ***** ok: [10.42.2.6] TASK [k3s/master : Register node-token file access mode] ************************************************************************************************************************** Thursday 10 November 2022 23:20:29 +0000 (0:00:00.458) 0:00:26.483 ***** ok: [10.42.2.6] TASK [k3s/master : Change file access node-token] ********************************************************************************************************************************* Thursday 10 November 2022 23:20:30 +0000 (0:00:00.329) 0:00:26.812 ***** changed: [10.42.2.6] TASK [k3s/master : Read node-token from master] *********************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.422) 0:00:27.234 ***** ok: [10.42.2.6] TASK [k3s/master : Store Master node-token] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.303) 0:00:27.537 ***** ok: [10.42.2.6] TASK [k3s/master : Restore node-token file access] ******************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.041) 0:00:27.579 ***** changed: [10.42.2.6] TASK [k3s/master : Create directory .kube] **************************************************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.232) 0:00:27.812 ***** changed: [10.42.2.6] TASK [k3s/master : Copy config file to user home directory] *********************************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.310) 0:00:28.122 ***** changed: [10.42.2.6] TASK [k3s/master : Replace https://localhost:6443 by https://master-ip:6443] ****************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.211) 0:00:28.334 ***** changed: [10.42.2.6] TASK [k3s/master : Create kubectl symlink] **************************************************************************************************************************************** Thursday 10 November 2022 23:20:32 +0000 (0:00:00.693) 0:00:29.027 ***** changed: [10.42.2.6] TASK [k3s/master : Create crictl symlink] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:32 +0000 (0:00:00.187) 0:00:29.215 ***** changed: [10.42.2.6] PLAY [node] *********************************************************************************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:32 +0000 (0:00:00.209) 0:00:29.424 ***** ok: [10.42.1.9] ok: [10.42.0.7] TASK [k3s/node : Copy K3s service file] ******************************************************************************************************************************************* Thursday 10 November 2022 23:20:33 +0000 (0:00:00.724) 0:00:30.149 ***** changed: [10.42.1.9] changed: [10.42.0.7] TASK [k3s/node : Enable and check K3s service] ************************************************************************************************************************************ Thursday 10 November 2022 23:20:34 +0000 (0:00:00.538) 0:00:30.687 ***** changed: [10.42.1.9] changed: [10.42.0.7] PLAY RECAP ************************************************************************************************************************************************************************ 10.42.0.7 : ok=8 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 10.42.1.9 : ok=8 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 10.42.2.6 : ok=19 changed=11 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 Thursday 10 November 2022 23:20:44 +0000 (0:00:10.395) 0:00:41.082 ***** =============================================================================== k3s/master : Enable and check K3s service --------------------------------------------------------------------------------------------------------------------------------- 11.16s k3s/node : Enable and check K3s service ----------------------------------------------------------------------------------------------------------------------------------- 10.40s download : Download k3s binary x64 ---------------------------------------------------------------------------------------------------------------------------------------- 10.04s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.54s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.72s k3s/master : Replace https://localhost:6443 by https://master-ip:6443 ------------------------------------------------------------------------------------------------------ 0.69s k3s/master : Copy K3s service file ----------------------------------------------------------------------------------------------------------------------------------------- 0.69s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.66s k3s/node : Copy K3s service file ------------------------------------------------------------------------------------------------------------------------------------------- 0.54s k3s/master : Wait for node-token ------------------------------------------------------------------------------------------------------------------------------------------- 0.46s k3s/master : Change file access node-token --------------------------------------------------------------------------------------------------------------------------------- 0.42s raspberrypi : Test for raspberry pi /proc/cpuinfo -------------------------------------------------------------------------------------------------------------------------- 0.36s k3s/master : Register node-token file access mode -------------------------------------------------------------------------------------------------------------------------- 0.33s prereq : Enable IPv4 forwarding -------------------------------------------------------------------------------------------------------------------------------------------- 0.32s k3s/master : Create directory .kube ---------------------------------------------------------------------------------------------------------------------------------------- 0.31s k3s/master : Read node-token from master ----------------------------------------------------------------------------------------------------------------------------------- 0.30s raspberrypi : Test for raspberry pi /proc/device-tree/model ---------------------------------------------------------------------------------------------------------------- 0.26s k3s/master : Restore node-token file access -------------------------------------------------------------------------------------------------------------------------------- 0.23s k3s/master : Copy config file to user home directory ----------------------------------------------------------------------------------------------------------------------- 0.21s k3s/master : Create crictl symlink ----------------------------------------------------------------------------------------------------------------------------------------- 0.21s

Plus there is a pretty good ansible playbook https://github.com/k3s-io/k3s-ansible

We have develop ansible role to do that. We did not put it as opensource yet. But there is one in the comunity which looks good: https://github.com/k3s-io/k3s-ansible

For the Ansible part, have a look at techno-tim/k3s-ansible for a basic k3s (not the full thing) with MetalLB ansible setup. There's also the original one (based on Traefik): k3s-io/k3s-ansible. Modfiy hosts and vars accordingly to your needs. Run your terraform plan, then run the ansible playbook once you've modified the vars accordingly. They're good starters, but for a full blown k8s ansible recipe, I'll be able to help you when I come back from holidays :) (16th of August).