k3s VS nerdctl

Opinionated meaning it picks, install, patches your CNI/Ingress/Load Balancer/DNS Server/Metrics Server/Monitoring Setup.

k3s is probably most well known as it ships with bunch of preinstall software: https://github.com/k3s-io/k3s so you can just start throwing yaml files at cluster and handling workloads. It's what I use for my homelab.

Paid things I've heard of include OpenStack and SideroLabs. Haven't used personally by SRE coworkers say good things about them.

For self-hosting I've found https://k3s.io to be really good from the SUSE people. Works on basically any Linux distro and makes self-hosting k8s not miserable.

Yes it’s going to depend on which k8s distribution you’re using. We have work in-progress for k3s to natively support nix-snapshotter: https://github.com/k3s-io/k3s/pull/9319

For other distributions, nix-snapshotter works with official containerd releases so it’s just a matter of toml configuration and a systemd unit for nix-snapshotter.

We run Kubernetes outside of NixOS, but yes the NixOS modules provided by the nix-snapshotter certainly make it simple.

K3S: is a lightweight distribution of Kubernetes that is designed for resource-constrained environments. It is an excellent option for running Kubernetes on a virtual machine or cloud server.

I recently purchased a used Lenovo M900 Think Centre (i7 with 32GB RAM) from eBay to expand my mini-homelab, which was just a single Synology DS218+ plugged into my ISP's router (yuck!). Since I've been spending a big chunk of time at work playing around with Kubernetes, I figured that I'd put my skills to the test and run a k3s node on the new server. While I was familiar with k3s before starting this project, I'd never actually run it before, opting for tools like kind (and minikube before that) to run small test clusters for my local development work.

Kubernetes - k3s distribution

I’ve created a local cluster with K3S and installing Windmill could not be simpler with just one chart to configure, which already has sane defaults to get started. For this demo we will also configure workers to passthrough environment variables to our scripts so that they have access to the Kubernetes API server for later.

You should be familiar with Kubernetes and have set up a Kubernetes cluster. I recommend k3s.

What about the docker-cli? colima also ships with a docker-compatible cli to interact with containerd called nerdctl. We can execute the same docker cli commands like:

Using nerdctl: https://github.com/containerd/nerdctl

I'd really disagree that compose files are somehow one-shot, or blindly modified. To the contrary, really, we have them checked in with the source code. Upon deployment to the cluster, the (running) services will be intelligently updated or replaced (in a rolling manner, causing zero downtime). LXC might be more elegant, but I have no idea what simple, file-based format I could use to let engineers describe the environment their app should run in without compose.

I need something that even junior devs can start up with a single command, that can be placed in the VCS along with the code, and that will not require deep Linux knowledge to get running. Open for suggestions here, really.

Now since Kubernetes works off of containerd I'll be taking a different approach on handling container builds by using nerdctl and the buildkit that comes bundled with it. I'll do this on the amd64 control plane node since it's beefier than my Raspberry Pi workers for handling builds and build related services. Go ahead and download and unpack the latest nerdctl release as of writing (make sure to check the release page in case there's a new one):

That's why nerdctl, their cli binary, is so well named.

Well, it is indeed a good option. However, containerd is a good alternative that is growing even among developers. Please see: https://github.com/containerd/nerdctl

Nerdctl/containerd has IPFS support :)

https://github.com/containerd/nerdctl/blob/main/docs/ipfs.md

nerdctl supports IPFS for both image pulling and pushing, including encrypted images and eStargz lazy pulling. For building, the current method is a locally hosted translator so that the traditional pulls can be converted to work over IPFS. They even have docs on running it on k8s node, though if my reading is correct this isn't exactly a cloud native approach (running systemd services on each node...).