k3s-ansible VS k3d

If you're down for some purchases, I recommend wholeheartedly to get the turing pi https://turingpi.com/ some raspberry modules and k3s-ansible https://github.com/k3s-io/k3s-ansible

I tried k3s recently and installing it was easy using k3s-ansible https://github.com/k3s-io/k3s-ansible

I use some ansible playbooks and when I am ready to update the version I update the variable in the playbooks. https://github.com/k3s-io/k3s-ansible

root@k3s1:~# git clone https://github.com/k3s-io/k3s-ansible Cloning into 'k3s-ansible'... remote: Enumerating objects: 922, done. remote: Total 922 (delta 0), reused 0 (delta 0), pack-reused 922 Receiving objects: 100% (922/922), 116.25 KiB | 3.32 MiB/s, done. Resolving deltas: 100% (351/351), done. root@k3s1:~# cd k3s-ansible/ root@k3s1:~/k3s-ansible# cp -R inventory/sample inventory/my-cluster root@k3s1:~/k3s-ansible# cat inventory/my-cluster/group_vars/all.yml --- k3s_version: v1.24.7+k3s1 ansible_user: ubuntu systemd_dir: /etc/systemd/system master_ip: "{{ hostvars[groups['master'][0]]['ansible_host'] | default(groups['master'][0]) }}" extra_server_args: "" extra_agent_args: "" root@k3s1:~/k3s-ansible# cat inventory/my-cluster/hosts.ini [master] 10.42.2.6 [node] 10.42.0.7 10.42.1.9 [k3s_cluster:children] master node root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# ssh-copy-id [email protected] /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added. root@k3s1:~/k3s-ansible# apt install python3-pip -y && pip install ansible root@k3s1:~/k3s-ansible# ansible-playbook site.yml -i inventory/my-cluster/hosts.ini PLAY [k3s_cluster] **************************************************************************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:03 +0000 (0:00:00.013) 0:00:00.013 ***** ok: [10.42.2.6] ok: [10.42.1.9] ok: [10.42.0.7] TASK [prereq : Set SELinux to disabled state] ************************************************************************************************************************************* Thursday 10 November 2022 23:20:04 +0000 (0:00:01.535) 0:00:01.548 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Enable IPv4 forwarding] ******************************************************************************************************************************************** Thursday 10 November 2022 23:20:04 +0000 (0:00:00.058) 0:00:01.607 ***** changed: [10.42.2.6] changed: [10.42.1.9] changed: [10.42.0.7] TASK [prereq : Enable IPv6 forwarding] ******************************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.318) 0:00:01.925 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Add br_netfilter to /etc/modules-load.d/] ************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.053) 0:00:01.978 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Load br_netfilter] ************************************************************************************************************************************************* Thursday 10 November 2022 23:20:05 +0000 (0:00:00.051) 0:00:02.030 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [prereq : Set bridge-nf-call-iptables (just to be sure)] ********************************************************************************************************************* Thursday 10 November 2022 23:20:05 +0000 (0:00:00.052) 0:00:02.082 ***** skipping: [10.42.2.6] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.2.6] => (item=net.bridge.bridge-nf-call-ip6tables) skipping: [10.42.0.7] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.0.7] => (item=net.bridge.bridge-nf-call-ip6tables) skipping: [10.42.1.9] => (item=net.bridge.bridge-nf-call-iptables) skipping: [10.42.1.9] => (item=net.bridge.bridge-nf-call-ip6tables) TASK [prereq : Add /usr/local/bin to sudo secure_path] **************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.072) 0:00:02.154 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [download : Download k3s binary x64] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:05 +0000 (0:00:00.103) 0:00:02.258 ***** [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually changed: [10.42.2.6] changed: [10.42.0.7] changed: [10.42.1.9] TASK [download : Download k3s binary arm64] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:10.036) 0:00:12.295 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [download : Download k3s binary armhf] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:00.067) 0:00:12.362 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Test for raspberry pi /proc/cpuinfo] ************************************************************************************************************************** Thursday 10 November 2022 23:20:15 +0000 (0:00:00.084) 0:00:12.447 ***** ok: [10.42.2.6] ok: [10.42.0.7] ok: [10.42.1.9] TASK [raspberrypi : Test for raspberry pi /proc/device-tree/model] **************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.359) 0:00:12.806 ***** ok: [10.42.2.6] ok: [10.42.0.7] ok: [10.42.1.9] TASK [raspberrypi : Set raspberry_pi fact to true] ******************************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.255) 0:00:13.062 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution to Raspbian] ************************************************************************************************************************ Thursday 10 November 2022 23:20:16 +0000 (0:00:00.060) 0:00:13.123 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution to Raspbian (ARM64 on Debian Buster)] *********************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.112) 0:00:13.235 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : Set detected_distribution_major_version] ********************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.073) 0:00:13.309 ***** skipping: [10.42.2.6] skipping: [10.42.0.7] skipping: [10.42.1.9] TASK [raspberrypi : execute OS related tasks on the Raspberry Pi] ***************************************************************************************************************** Thursday 10 November 2022 23:20:16 +0000 (0:00:00.080) 0:00:13.389 ***** skipping: [10.42.2.6] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) skipping: [10.42.0.7] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) skipping: [10.42.1.9] => (item=/root/k3s-ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml) PLAY [master] ********************************************************************************************************************************************************************* TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:16 +0000 (0:00:00.125) 0:00:13.515 ***** ok: [10.42.2.6] TASK [k3s/master : Copy K3s service file] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:17 +0000 (0:00:00.660) 0:00:14.175 ***** changed: [10.42.2.6] TASK [k3s/master : Enable and check K3s service] ********************************************************************************************************************************** Thursday 10 November 2022 23:20:18 +0000 (0:00:00.688) 0:00:14.864 ***** changed: [10.42.2.6] TASK [k3s/master : Wait for node-token] ******************************************************************************************************************************************* Thursday 10 November 2022 23:20:29 +0000 (0:00:11.160) 0:00:26.025 ***** ok: [10.42.2.6] TASK [k3s/master : Register node-token file access mode] ************************************************************************************************************************** Thursday 10 November 2022 23:20:29 +0000 (0:00:00.458) 0:00:26.483 ***** ok: [10.42.2.6] TASK [k3s/master : Change file access node-token] ********************************************************************************************************************************* Thursday 10 November 2022 23:20:30 +0000 (0:00:00.329) 0:00:26.812 ***** changed: [10.42.2.6] TASK [k3s/master : Read node-token from master] *********************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.422) 0:00:27.234 ***** ok: [10.42.2.6] TASK [k3s/master : Store Master node-token] *************************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.303) 0:00:27.537 ***** ok: [10.42.2.6] TASK [k3s/master : Restore node-token file access] ******************************************************************************************************************************** Thursday 10 November 2022 23:20:30 +0000 (0:00:00.041) 0:00:27.579 ***** changed: [10.42.2.6] TASK [k3s/master : Create directory .kube] **************************************************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.232) 0:00:27.812 ***** changed: [10.42.2.6] TASK [k3s/master : Copy config file to user home directory] *********************************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.310) 0:00:28.122 ***** changed: [10.42.2.6] TASK [k3s/master : Replace https://localhost:6443 by https://master-ip:6443] ****************************************************************************************************** Thursday 10 November 2022 23:20:31 +0000 (0:00:00.211) 0:00:28.334 ***** changed: [10.42.2.6] TASK [k3s/master : Create kubectl symlink] **************************************************************************************************************************************** Thursday 10 November 2022 23:20:32 +0000 (0:00:00.693) 0:00:29.027 ***** changed: [10.42.2.6] TASK [k3s/master : Create crictl symlink] ***************************************************************************************************************************************** Thursday 10 November 2022 23:20:32 +0000 (0:00:00.187) 0:00:29.215 ***** changed: [10.42.2.6] PLAY [node] *********************************************************************************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************************************************************************ Thursday 10 November 2022 23:20:32 +0000 (0:00:00.209) 0:00:29.424 ***** ok: [10.42.1.9] ok: [10.42.0.7] TASK [k3s/node : Copy K3s service file] ******************************************************************************************************************************************* Thursday 10 November 2022 23:20:33 +0000 (0:00:00.724) 0:00:30.149 ***** changed: [10.42.1.9] changed: [10.42.0.7] TASK [k3s/node : Enable and check K3s service] ************************************************************************************************************************************ Thursday 10 November 2022 23:20:34 +0000 (0:00:00.538) 0:00:30.687 ***** changed: [10.42.1.9] changed: [10.42.0.7] PLAY RECAP ************************************************************************************************************************************************************************ 10.42.0.7 : ok=8 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 10.42.1.9 : ok=8 changed=4 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 10.42.2.6 : ok=19 changed=11 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0 Thursday 10 November 2022 23:20:44 +0000 (0:00:10.395) 0:00:41.082 ***** =============================================================================== k3s/master : Enable and check K3s service --------------------------------------------------------------------------------------------------------------------------------- 11.16s k3s/node : Enable and check K3s service ----------------------------------------------------------------------------------------------------------------------------------- 10.40s download : Download k3s binary x64 ---------------------------------------------------------------------------------------------------------------------------------------- 10.04s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.54s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.72s k3s/master : Replace https://localhost:6443 by https://master-ip:6443 ------------------------------------------------------------------------------------------------------ 0.69s k3s/master : Copy K3s service file ----------------------------------------------------------------------------------------------------------------------------------------- 0.69s Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.66s k3s/node : Copy K3s service file ------------------------------------------------------------------------------------------------------------------------------------------- 0.54s k3s/master : Wait for node-token ------------------------------------------------------------------------------------------------------------------------------------------- 0.46s k3s/master : Change file access node-token --------------------------------------------------------------------------------------------------------------------------------- 0.42s raspberrypi : Test for raspberry pi /proc/cpuinfo -------------------------------------------------------------------------------------------------------------------------- 0.36s k3s/master : Register node-token file access mode -------------------------------------------------------------------------------------------------------------------------- 0.33s prereq : Enable IPv4 forwarding -------------------------------------------------------------------------------------------------------------------------------------------- 0.32s k3s/master : Create directory .kube ---------------------------------------------------------------------------------------------------------------------------------------- 0.31s k3s/master : Read node-token from master ----------------------------------------------------------------------------------------------------------------------------------- 0.30s raspberrypi : Test for raspberry pi /proc/device-tree/model ---------------------------------------------------------------------------------------------------------------- 0.26s k3s/master : Restore node-token file access -------------------------------------------------------------------------------------------------------------------------------- 0.23s k3s/master : Copy config file to user home directory ----------------------------------------------------------------------------------------------------------------------- 0.21s k3s/master : Create crictl symlink ----------------------------------------------------------------------------------------------------------------------------------------- 0.21s

Plus there is a pretty good ansible playbook https://github.com/k3s-io/k3s-ansible

We have develop ansible role to do that. We did not put it as opensource yet. But there is one in the comunity which looks good: https://github.com/k3s-io/k3s-ansible

For the Ansible part, have a look at techno-tim/k3s-ansible for a basic k3s (not the full thing) with MetalLB ansible setup. There's also the original one (based on Traefik): k3s-io/k3s-ansible. Modfiy hosts and vars accordingly to your needs. Run your terraform plan, then run the ansible playbook once you've modified the vars accordingly. They're good starters, but for a full blown k8s ansible recipe, I'll be able to help you when I come back from holidays :) (16th of August).

K3D: is a lightweight distribution of Kubernetes designed for resource-constrained environments. It is an excellent option for running Kubernetes on virtual machines or cloud servers.

k3d is a lightweight wrapper that makes running Kubernetes (specifically, the lightweight k3s distribution) in Docker straightforward and efficient. It's designed to provide developers with a quick and easy way to test Kubernetes without the overhead of setting up a full cluster.

If you customer prefers to run the standard docker engine you could use k3d

You don’t need to run Rancher from a Kubernetes cluster, the rancher/rancher image works fine with Docker (it uses k3d, aka « k3s in docker » : https://k3d.io/).

K3d and Skaffold for local development

If you are a developer and want to learn how to deploy applications to a cluster, getting a cluster up an running can be a daunting task in it's own rights. There are many ways to do it: spinning up local virtual machines and configuring from scratch or using tools like minikube, etc. You may not care for the pain of setting up and configuring a cluster, and if that is you, then the quickest way that I have found is using k3d.