kubespray VS deckhouse

When it came to deployment, I had several options, and I chose the hard way: deploying Kubernetes on bare metal nodes using KubeSpray. Troubleshooting bare metal Kubernetes deployments honed my skills in pinpointing issues. This hands-on experience provided a deep understanding of how each component, like the Control Plane, kubelet, Container Runtime, and scheduler, interacts to orchestrate containers.

Worth noting that if you like ansible, Kubespray has had documented air-gap installation since 2018 https://github.com/kubernetes-sigs/kubespray/commit/963c3479...

calico v3.7.3 kubernetes v1.16.0 installed via kubespray https://github.com/kubernetes-sigs/kubespray

About 1yr ago I ran k8s with 300 nodes using kube spray https://github.com/kubernetes-sigs/kubespray . Never had any real issue with it. We did finally move to the cloud though.

I would highly recommend https://github.com/kubernetes-sigs/kubespray

We build our own distro called Compliant Kubernetes, and we use (a fork of) kubespray to install the base Kubernetes layer. Our distro is entirely open source, so you can use it as a reference, if you want.

If you want to run Kubernetes on VM's you can look into kubespray and play around with the options for installing and checking the configuration after the fact.

And while k3s sounds easy, it's not after even a slightly larger scale.

If one willing to have in-house k8s today I would recommend https://deckhouse.io/ (I'm not affiliated with them)

Deckhouse

Check Deckhouse as well (https://github.com/deckhouse/deckhouse). Cilium integration was added there recently.

Release Deckhouse v1.26.0 · deckhouse/deckhouse (github.com)

Consider Deckhouse as another option that can be installed everywhere: bare metal, private and public clouds. It has an Open Source core offered in Community Edition, but the pricing for Enterprise Edition is also a fit for small companies. All configuration is made via Custom Resources, all routines (like updating Kubernetes versions and related system components) are automated.

We use cert-manager in our K8s platform for years and it works perfectly. I don't think there is any chance today to compete with it in terms of community adoption. I also can't find any GitHub links for your project which is the most widely accepted way to become trusted by the community — simply because it covers all the basic needs everyone got used to (in the Open Source world): we can see how the code is developed, we can contribute to it, discuss the issues and concerns we have, etc.

We use Vector to ship logs to different storages (including Loki). You can even find the implementation here, however, I am not sure how useful it might be since it's quite specific (Golang hooks for addon-operator). Our manifests are available there as well.