kubefire VS k3s

root@kubefire:~# kubefire install INFO[2022-11-11T11:46:13Z] downloading https://raw.githubusercontent.com/innobead/kubefire/v0.3.8/scripts/install-prerequisites.sh to save /root/.kubefire/bin/v0.3.8/install-prerequisites.sh force=false version=v0.3.8 INFO[2022-11-11T11:46:14Z] running script (install-prerequisites.sh) version=v0.3.8 INFO[2022-11-11T11:46:14Z] running /root/.kubefire/bin/v0.3.8/install-prerequisites.sh version=v0.3.8 INFO[2022-11-11T11:46:14Z] + TMP_DIR=/tmp/kubefire INFO[2022-11-11T11:46:14Z] ++ go env GOARCH INFO[2022-11-11T11:46:14Z] ++ echo amd64 INFO[2022-11-11T11:46:14Z] + GOARCH=amd64 INFO[2022-11-11T11:46:14Z] + KUBEFIRE_VERSION=v0.3.8 INFO[2022-11-11T11:46:14Z] + CONTAINERD_VERSION=v1.6.6 + IGNITE_VERION=v0.10.0 INFO[2022-11-11T11:46:14Z] + CNI_VERSION=v1.1.1 + RUNC_VERSION=v1.1.3 INFO[2022-11-11T11:46:14Z] + '[' -z v0.3.8 ']' + '[' -z v1.6.6 ']' + '[' -z v0.10.0 ']' + '[' -z v1.1.1 ']' + '[' -z v1.1.3 ']' INFO[2022-11-11T11:46:14Z] ++ sed -E 's/(v[0-9]+\.[0-9]+\.[0-9]+)[a-zA-Z0-9\-]*/\1/g' INFO[2022-11-11T11:46:14Z] +++ echo v0.3.8 INFO[2022-11-11T11:46:14Z] + STABLE_KUBEFIRE_VERSION=v0.3.8 INFO[2022-11-11T11:46:14Z] + rm -rf /tmp/kubefire INFO[2022-11-11T11:46:14Z] + mkdir -p /tmp/kubefire INFO[2022-11-11T11:46:14Z] + pushd /tmp/kubefire /tmp/kubefire /root INFO[2022-11-11T11:46:14Z] + trap cleanup EXIT ERR INT TERM INFO[2022-11-11T11:46:14Z] + check_virtualization + _is_arm_arch INFO[2022-11-11T11:46:14Z] + uname -m INFO[2022-11-11T11:46:14Z] + grep aarch64 INFO[2022-11-11T11:46:14Z] + return 1 INFO[2022-11-11T11:46:14Z] + lscpu INFO[2022-11-11T11:46:14Z] + grep 'Virtuali[s|z]ation' INFO[2022-11-11T11:46:14Z] Virtualization: VT-x Virtualization type: full INFO[2022-11-11T11:46:14Z] + lsmod INFO[2022-11-11T11:46:14Z] + grep kvm INFO[2022-11-11T11:46:14Z] kvm_intel 372736 0 kvm 1028096 1 kvm_intel INFO[2022-11-11T11:46:14Z] + install_runc + _check_version /usr/local/bin/runc -version v1.1.3 INFO[2022-11-11T11:46:14Z] + set +o pipefail + local exec_name=/usr/local/bin/runc + local exec_version_cmd=-version + local version=v1.1.3 + command -v /usr/local/bin/runc + return 1 + _is_arm_arch INFO[2022-11-11T11:46:14Z] + uname -m INFO[2022-11-11T11:46:14Z] + grep aarch64 INFO[2022-11-11T11:46:14Z] + return 1 INFO[2022-11-11T11:46:14Z] + curl -sfSL https://github.com/opencontainers/runc/releases/download/v1.1.3/runc.amd64 -o runc INFO[2022-11-11T11:46:14Z] + chmod +x runc INFO[2022-11-11T11:46:14Z] + sudo mv runc /usr/local/bin/ INFO[2022-11-11T11:46:14Z] + install_containerd + _check_version /usr/local/bin/containerd --version v1.6.6 INFO[2022-11-11T11:46:14Z] + set +o pipefail + local exec_name=/usr/local/bin/containerd + local exec_version_cmd=--version + local version=v1.6.6 + command -v /usr/local/bin/containerd + return 1 + local version=1.6.6 + local dir=containerd-1.6.6 + _is_arm_arch INFO[2022-11-11T11:46:14Z] + uname -m INFO[2022-11-11T11:46:14Z] + grep aarch64 INFO[2022-11-11T11:46:14Z] + return 1 INFO[2022-11-11T11:46:14Z] + curl -sfSLO https://github.com/containerd/containerd/releases/download/v1.6.6/containerd-1.6.6-linux-amd64.tar.gz INFO[2022-11-11T11:46:15Z] + mkdir -p containerd-1.6.6 INFO[2022-11-11T11:46:15Z] + tar -zxvf containerd-1.6.6-linux-amd64.tar.gz -C containerd-1.6.6 INFO[2022-11-11T11:46:15Z] bin/ bin/containerd-shim INFO[2022-11-11T11:46:15Z] bin/containerd INFO[2022-11-11T11:46:16Z] bin/containerd-shim-runc-v1 INFO[2022-11-11T11:46:16Z] bin/containerd-stress INFO[2022-11-11T11:46:16Z] bin/containerd-shim-runc-v2 INFO[2022-11-11T11:46:16Z] bin/ctr INFO[2022-11-11T11:46:17Z] + chmod +x containerd-1.6.6/bin/containerd containerd-1.6.6/bin/containerd-shim containerd-1.6.6/bin/containerd-shim-runc-v1 containerd-1.6.6/bin/containerd-shim-runc-v2 containerd-1.6.6/bin/containerd-stress containerd-1.6.6/bin/ctr INFO[2022-11-11T11:46:17Z] + sudo mv containerd-1.6.6/bin/containerd containerd-1.6.6/bin/containerd-shim containerd-1.6.6/bin/containerd-shim-runc-v1 containerd-1.6.6/bin/containerd-shim-runc-v2 containerd-1.6.6/bin/containerd-stress containerd-1.6.6/bin/ctr /usr/local/bin/ INFO[2022-11-11T11:46:17Z] + curl -sfSLO https://raw.githubusercontent.com/containerd/containerd/v1.6.6/containerd.service INFO[2022-11-11T11:46:17Z] + sudo groupadd containerd INFO[2022-11-11T11:46:17Z] + sudo mv containerd.service /etc/systemd/system/containerd.service INFO[2022-11-11T11:46:17Z] ++ command -v chgrp INFO[2022-11-11T11:46:17Z] ++ tr -d '\n' INFO[2022-11-11T11:46:17Z] + chgrp_path=/usr/bin/chgrp INFO[2022-11-11T11:46:17Z] + sudo sed -i -E 's#(ExecStart=/usr/local/bin/containerd)#\1\nExecStartPost=/usr/bin/chgrp containerd /run/containerd/containerd.sock#g' /etc/systemd/system/containerd.service INFO[2022-11-11T11:46:17Z] + sudo mkdir -p /etc/containerd INFO[2022-11-11T11:46:17Z] + containerd config default INFO[2022-11-11T11:46:17Z] + sudo tee /etc/containerd/config.toml INFO[2022-11-11T11:46:17Z] + sudo systemctl enable --now containerd INFO[2022-11-11T11:46:17Z] Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /etc/systemd/system/containerd.service. INFO[2022-11-11T11:46:17Z] + install_cni + _check_version /opt/cni/bin/bridge --version v1.1.1 + set +o pipefail INFO[2022-11-11T11:46:17Z] + local exec_name=/opt/cni/bin/bridge + local exec_version_cmd=--version + local version=v1.1.1 + command -v /opt/cni/bin/bridge INFO[2022-11-11T11:46:17Z] + return 1 INFO[2022-11-11T11:46:17Z] + mkdir -p /opt/cni/bin INFO[2022-11-11T11:46:17Z] + local f=https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz + _is_arm_arch INFO[2022-11-11T11:46:17Z] + uname -m INFO[2022-11-11T11:46:17Z] + grep aarch64 INFO[2022-11-11T11:46:17Z] + return 1 INFO[2022-11-11T11:46:17Z] + curl -sfSL https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz INFO[2022-11-11T11:46:17Z] + tar -C /opt/cni/bin -xz INFO[2022-11-11T11:46:19Z] + install_cni_patches + _is_arm_arch INFO[2022-11-11T11:46:19Z] + uname -m INFO[2022-11-11T11:46:19Z] + grep aarch64 INFO[2022-11-11T11:46:19Z] + return 1 + curl -o host-local-rev -sfSL https://github.com/innobead/kubefire/releases/download/v0.3.8/host-local-rev-linux-amd64 INFO[2022-11-11T11:46:19Z] + chmod +x host-local-rev INFO[2022-11-11T11:46:19Z] + sudo mv host-local-rev /opt/cni/bin/ INFO[2022-11-11T11:46:19Z] + install_ignite + _check_version /usr/local/bin/ignite version v0.10.0 + set +o pipefail INFO[2022-11-11T11:46:19Z] + local exec_name=/usr/local/bin/ignite + local exec_version_cmd=version + local version=v0.10.0 + command -v /usr/local/bin/ignite + return 1 INFO[2022-11-11T11:46:19Z] + for binary in ignite ignited + echo 'Installing ignite...' INFO[2022-11-11T11:46:19Z] Installing ignite... INFO[2022-11-11T11:46:19Z] + local f=https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignite-amd64 + _is_arm_arch INFO[2022-11-11T11:46:19Z] + uname -m INFO[2022-11-11T11:46:19Z] + grep aarch64 INFO[2022-11-11T11:46:19Z] + return 1 + curl -sfSLo ignite https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignite-amd64 INFO[2022-11-11T11:46:20Z] + chmod +x ignite INFO[2022-11-11T11:46:20Z] + sudo mv ignite /usr/local/bin INFO[2022-11-11T11:46:20Z] + for binary in ignite ignited + echo 'Installing ignited...' Installing ignited... + local f=https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignited-amd64 INFO[2022-11-11T11:46:20Z] + _is_arm_arch INFO[2022-11-11T11:46:20Z] + grep aarch64 + uname -m INFO[2022-11-11T11:46:20Z] + return 1 + curl -sfSLo ignited https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignited-amd64 INFO[2022-11-11T11:46:21Z] + chmod +x ignited INFO[2022-11-11T11:46:21Z] + sudo mv ignited /usr/local/bin INFO[2022-11-11T11:46:21Z] + check_ignite + ignite version INFO[2022-11-11T11:46:21Z] Ignite version: version.Info{Major:"0", Minor:"10", GitVersion:"v0.10.0", GitCommit:"4540abeb9ba6daba32a72ef2b799095c71ebacb0", GitTreeState:"clean", BuildDate:"2021-07-19T20:52:59Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64", SandboxImage:version.Image{Name:"weaveworks/ignite", Tag:"v0.10.0", Delimeter:":"}, KernelImage:version.Image{Name:"weaveworks/ignite-kernel", Tag:"5.10.51", Delimeter:":"}} INFO[2022-11-11T11:46:21Z] Firecracker version: v0.22.4 INFO[2022-11-11T11:46:21Z] + create_cni_default_config INFO[2022-11-11T11:46:21Z] + mkdir -p /etc/cni/net.d/ INFO[2022-11-11T11:46:21Z] + sudo cat INFO[2022-11-11T11:46:21Z] + popd /root + cleanup INFO[2022-11-11T11:46:21Z] + rm -rf /tmp/kubefire

Opinionated meaning it picks, install, patches your CNI/Ingress/Load Balancer/DNS Server/Metrics Server/Monitoring Setup.

k3s is probably most well known as it ships with bunch of preinstall software: https://github.com/k3s-io/k3s so you can just start throwing yaml files at cluster and handling workloads. It's what I use for my homelab.

Paid things I've heard of include OpenStack and SideroLabs. Haven't used personally by SRE coworkers say good things about them.

For self-hosting I've found https://k3s.io to be really good from the SUSE people. Works on basically any Linux distro and makes self-hosting k8s not miserable.

Yes it’s going to depend on which k8s distribution you’re using. We have work in-progress for k3s to natively support nix-snapshotter: https://github.com/k3s-io/k3s/pull/9319

For other distributions, nix-snapshotter works with official containerd releases so it’s just a matter of toml configuration and a systemd unit for nix-snapshotter.

We run Kubernetes outside of NixOS, but yes the NixOS modules provided by the nix-snapshotter certainly make it simple.

K3S: is a lightweight distribution of Kubernetes that is designed for resource-constrained environments. It is an excellent option for running Kubernetes on a virtual machine or cloud server.

I recently purchased a used Lenovo M900 Think Centre (i7 with 32GB RAM) from eBay to expand my mini-homelab, which was just a single Synology DS218+ plugged into my ISP's router (yuck!). Since I've been spending a big chunk of time at work playing around with Kubernetes, I figured that I'd put my skills to the test and run a k3s node on the new server. While I was familiar with k3s before starting this project, I'd never actually run it before, opting for tools like kind (and minikube before that) to run small test clusters for my local development work.

Kubernetes - k3s distribution

I’ve created a local cluster with K3S and installing Windmill could not be simpler with just one chart to configure, which already has sane defaults to get started. For this demo we will also configure workers to passthrough environment variables to our scripts so that they have access to the Kubernetes API server for later.

You should be familiar with Kubernetes and have set up a Kubernetes cluster. I recommend k3s.