k3s VS vcluster

Opinionated meaning it picks, install, patches your CNI/Ingress/Load Balancer/DNS Server/Metrics Server/Monitoring Setup.

k3s is probably most well known as it ships with bunch of preinstall software: https://github.com/k3s-io/k3s so you can just start throwing yaml files at cluster and handling workloads. It's what I use for my homelab.

Paid things I've heard of include OpenStack and SideroLabs. Haven't used personally by SRE coworkers say good things about them.

For self-hosting I've found https://k3s.io to be really good from the SUSE people. Works on basically any Linux distro and makes self-hosting k8s not miserable.

Yes it’s going to depend on which k8s distribution you’re using. We have work in-progress for k3s to natively support nix-snapshotter: https://github.com/k3s-io/k3s/pull/9319

For other distributions, nix-snapshotter works with official containerd releases so it’s just a matter of toml configuration and a systemd unit for nix-snapshotter.

We run Kubernetes outside of NixOS, but yes the NixOS modules provided by the nix-snapshotter certainly make it simple.

K3S: is a lightweight distribution of Kubernetes that is designed for resource-constrained environments. It is an excellent option for running Kubernetes on a virtual machine or cloud server.

I recently purchased a used Lenovo M900 Think Centre (i7 with 32GB RAM) from eBay to expand my mini-homelab, which was just a single Synology DS218+ plugged into my ISP's router (yuck!). Since I've been spending a big chunk of time at work playing around with Kubernetes, I figured that I'd put my skills to the test and run a k3s node on the new server. While I was familiar with k3s before starting this project, I'd never actually run it before, opting for tools like kind (and minikube before that) to run small test clusters for my local development work.

Kubernetes - k3s distribution

I’ve created a local cluster with K3S and installing Windmill could not be simpler with just one chart to configure, which already has sane defaults to get started. For this demo we will also configure workers to passthrough environment variables to our scripts so that they have access to the Kubernetes API server for later.

You should be familiar with Kubernetes and have set up a Kubernetes cluster. I recommend k3s.

Kubernetes? You mean the container orchestration system where they forgot to add Multi-tenancy? And no namespaces are not Multi-tenancy...

https://www.vcluster.com/

Whats the difference between this and vcluster (https://github.com/loft-sh/vcluster)?

Yep, as we see it they compliment each other quite well. DevPod takes your workspace to the cloud and DevSpace let's you develop against your Kubernetes cluster - potentially the same one you used to start your workspace.

Internally we use both in our development setup, spinning up remote workspaces using DevPod, installing DevSpace and kind into the devcontainer, then using DevSpace to develop against the cluster. See the vcluster setup[1] as an example

[1]https://github.com/loft-sh/vcluster/tree/main/.devcontainer

The tenants are internal dev teams so yeah maybe not. I was considering multi-tenanting different environments isolated at the kube layer with vCluster and have the vCluster pods running in Kata containers giving maximum isolation but still having a single management cluster. Ideally also avoiding the need to buy a second set of hardware for a dev environment

Vcluster

Looks interesting. How does this solution compare to vcluster?

sounds like the best option for you , is a tool called VCluster by loft ( https://www.vcluster.com/) , this way you can install as many k8s cluster as you want in the same k8s host cluster , those cluster share workers nodes and networking, but each has a separated "api server" , so it looks like you have a dedicated cluster with their own namespaces and tools . take a look at the docs to get a better understanding and how they work.

We're about to run a PoC with vcluster for isolated sandboxes, this might be relevant to you too

I think namespaces, RBAC and network policies are sufficient to partition users from the same organisation. I would investigate the use of vcluster ig you want to give your users even more isolation and capability (such as installing CRDs)

Depends on the use case. Namespaces provides soft isolation (so it means they share same Apiserver, PV's and global resources such as CRD's), but can be restricted with network policies. So it means, there's still potential in breaking other namespaces if you change PV's or CRD's which are used by other namespaces. Multi-Cluster solution can provide full isolation, but its also really expensive in resource consumption and maintenance/management effort. If namespaced-isolation isnt enough for your use case, you can consider vclusters (https://www.vcluster.com/)