kotal VS k3s

https://kotal.co spin nodes on your infrastructure and create tls secure endpoints for you.

root@edgevpn1:~# kubectl apply -f https://github.com/kotalco/kotal/releases/download/v0.1-alpha.6/kotal.yaml namespace/kotal created customresourcedefinition.apiextensions.k8s.io/beaconnodes.ethereum2.kotal.io created customresourcedefinition.apiextensions.k8s.io/clusterpeers.ipfs.kotal.io created customresourcedefinition.apiextensions.k8s.io/nodes.chainlink.kotal.io created customresourcedefinition.apiextensions.k8s.io/nodes.ethereum.kotal.io created customresourcedefinition.apiextensions.k8s.io/nodes.filecoin.kotal.io created customresourcedefinition.apiextensions.k8s.io/nodes.polkadot.kotal.io created customresourcedefinition.apiextensions.k8s.io/peers.ipfs.kotal.io created customresourcedefinition.apiextensions.k8s.io/validators.ethereum2.kotal.io created mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating-webhook-configuration created role.rbac.authorization.k8s.io/leader-election-role created clusterrole.rbac.authorization.k8s.io/manager-role created clusterrole.rbac.authorization.k8s.io/metrics-reader created clusterrole.rbac.authorization.k8s.io/proxy-role created rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/manager-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created service/controller-manager-metrics-service created service/webhook-service created deployment.apps/controller-manager created certificate.cert-manager.io/serving-cert created issuer.cert-manager.io/selfsigned-issuer created validatingwebhookconfiguration.admissionregistration.k8s.io/validating-webhook-configuration created root@edgevpn1:~# kubectl get po,svc -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system pod/local-path-provisioner-5b5579c644-pv9cj 1/1 Running 0 48m kube-system pod/coredns-75fc8f8fff-clcz6 1/1 Running 0 48m kube-system pod/helm-install-traefik-crd-tz9t9 0/1 Completed 0 48m kube-system pod/helm-install-traefik-bwc74 0/1 Completed 1 48m kube-system pod/svclb-traefik-eedbcbde-dj69n 2/2 Running 0 48m kube-system pod/traefik-9c6dc6686-nckxj 1/1 Running 0 48m kube-system pod/metrics-server-5c8978b444-ggqg7 1/1 Running 0 48m kube-system pod/svclb-traefik-eedbcbde-sb8h9 2/2 Running 0 33m cert-manager pod/cert-manager-77f9f95c5c-fk98d 1/1 Running 0 3m39s cert-manager pod/cert-manager-webhook-765b6f5ffc-fzj5p 1/1 Running 0 3m39s cert-manager pod/cert-manager-cainjector-769c877458-4wxmb 1/1 Running 2 (71s ago) 3m39s kotal pod/controller-manager-5784dc84d5-kbr58 2/2 Running 0 71s NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.43.0.1 443/TCP 48m kube-system service/kube-dns ClusterIP 10.43.0.10 53/UDP,53/TCP,9153/TCP 48m kube-system service/metrics-server ClusterIP 10.43.26.165 443/TCP 48m kube-system service/traefik LoadBalancer 10.43.243.18 10.1.0.3,10.1.0.4 80:31735/TCP,443:30749/TCP 48m cert-manager service/cert-manager ClusterIP 10.43.12.233 9402/TCP 3m39s cert-manager service/cert-manager-webhook ClusterIP 10.43.217.216 443/TCP 3m39s kotal service/controller-manager-metrics-service ClusterIP 10.43.188.129 8443/TCP 71s kotal service/webhook-service ClusterIP 10.43.142.211 443/TCP 71s

we're developing a Kubernetes operator https://github.com/kotalco/kotal and we're testing it on 1.19 to 1.25 k8s clusters using KinD, by creating cluster, testing, then tearing it down.

Opinionated meaning it picks, install, patches your CNI/Ingress/Load Balancer/DNS Server/Metrics Server/Monitoring Setup.

k3s is probably most well known as it ships with bunch of preinstall software: https://github.com/k3s-io/k3s so you can just start throwing yaml files at cluster and handling workloads. It's what I use for my homelab.

Paid things I've heard of include OpenStack and SideroLabs. Haven't used personally by SRE coworkers say good things about them.

For self-hosting I've found https://k3s.io to be really good from the SUSE people. Works on basically any Linux distro and makes self-hosting k8s not miserable.

Yes it’s going to depend on which k8s distribution you’re using. We have work in-progress for k3s to natively support nix-snapshotter: https://github.com/k3s-io/k3s/pull/9319

For other distributions, nix-snapshotter works with official containerd releases so it’s just a matter of toml configuration and a systemd unit for nix-snapshotter.

We run Kubernetes outside of NixOS, but yes the NixOS modules provided by the nix-snapshotter certainly make it simple.

K3S: is a lightweight distribution of Kubernetes that is designed for resource-constrained environments. It is an excellent option for running Kubernetes on a virtual machine or cloud server.

I recently purchased a used Lenovo M900 Think Centre (i7 with 32GB RAM) from eBay to expand my mini-homelab, which was just a single Synology DS218+ plugged into my ISP's router (yuck!). Since I've been spending a big chunk of time at work playing around with Kubernetes, I figured that I'd put my skills to the test and run a k3s node on the new server. While I was familiar with k3s before starting this project, I'd never actually run it before, opting for tools like kind (and minikube before that) to run small test clusters for my local development work.

Kubernetes - k3s distribution

I’ve created a local cluster with K3S and installing Windmill could not be simpler with just one chart to configure, which already has sane defaults to get started. For this demo we will also configure workers to passthrough environment variables to our scripts so that they have access to the Kubernetes API server for later.

You should be familiar with Kubernetes and have set up a Kubernetes cluster. I recommend k3s.