hcloud-cloud-controller-manager VS swarmsible

When it comes to load balancing, I think the hcloud-cloud-controller-manager[1] is probably your best bet, and although I haven't tested it, I'm sure it can be coerced into some kind of working configuration with the vSwitch/Cloud Network coupling, even none of cluster is actually Cloud-based.

I haven't used Sidero Omni yet, but if it's as well architected as Talos is, I'm sure it's an excellent solution. It still leaves open the question of ordering and provisioning the servers themselves. For simpler use-cases it wouldn't be too difficult to hack together a script to interact with the Hetzner Robot API to achieve this goal, but if I wanted any level of robustness, and if you'll excuse the shameless plug, I think I'd write a custom operator in Rust using my hrobot-rs[2] library :)

[1] https://github.com/hetznercloud/hcloud-cloud-controller-mana...

Did you actually use the Hetzner LBs or did you just send traffic to the NodePort?

It's not even close to major public cloud providers, but this is my setup:

* https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne... (Terraform, Kubernetes bootstrap)

* Flux for CI

* nginx-ingress + Hetzner Loadbalancer (thanks to https://github.com/hetznercloud/hcloud-cloud-controller-mana...)

* Hetzner storage volumes (thanks to https://github.com/hetznercloud/csi-driver)

Kube-Hetzner supports Hetzner Cloud loadbalancers and volumes out of the box, though it also supports other components.

I plan to do the same or have already implemented it. I use the Hetzner private networks with the hcloud-cloud-controller-manager https://github.com/hetznercloud/hcloud-cloud-controller-manager. This creates routes so that the nodes are also connected to each other in the private network. All kubernetes resources listen on the private network. Except for the Kubernetes API, which listens on the external IP of the master so I can use kubectl from home. The actual gateway into the cluster is controlled by a load balancer from Hetzner.

I want to use Hetzner's Load Balancer as a Nginx Ingress controller. I follow this suggestion to install Nginx Ingress via helm.

We use Docker Swarm for our deployments, so I will answer the questions based on that.

We have built some tooling around setting up and maintaining the swarm using ansible [0]. We also added some Hetzner flavour to that [1] which allows us to automatically spin up completely new clusters in a really short amount of time.

deploy from source repo:

- We use Azure DevOps pipelines that automate deployments based on environment configs living in an encrypted state in Git repos. We use [2] and [3] to make it easier to organize the deployments using `docker stack deploy` under the hood.

keep software up to date:

- We are currently looking into CVE scanners that export into prometheus to give us an idea of what we should update

load balancing:

- depending on the project, Hetzner LB or Cloudflare

handle scaling:

- manually, but i would love to build some autoscaler for swarm that interacts with our tooling [0] and [1]

automate backups:

- docker swarm cronjobs either via jobs with restart condition and a delay or [4]

maintain security:

- Hetzner LB is front facing. Communication is done via encrypted networks inside Hetzner private cloud networks

- [0] https://github.com/neuroforgede/swarmsible

Pretty much the same as our goto for projects at work: Hetzner + Docker (Swarm) with some Ansible to orchestrate things

We have built some automation around cluster management over at https://github.com/neuroforgede/swarmsible.

I used to do everything in ansible, but Docker Stacks are just so much nicer to use.

In any case automation is king. I don't have to remember stuff if I can just look at some IaC Code :).

Our tooling can be found here https://github.com/neuroforgede/swarmsible . It is not yet documented tbh, but most things are pretty straight forward to use if you have used ansible, docker etc already.

Story of one of the projects I am involved in:

We came from Ansible managed deployments of vanilla docker with nginx as single node ingress with another load balancer on top of that.

Worked fine, but HA for containers that are only allowed to exist once in the stack was one thing that caused us headaches.

Then, we had a workshop for Rancher RKE. Looked promising at the start, but operating it became a headache as we didn't have enough people in the project team to maintain it. Certificates expiring was an issue and the fact that you actually kinda had to baby-sit the cluster was a turn off.

We killed the switch to kubernetes.

In the meantime we were toying around with Docker Swarm for smaller scale deployments and inhouse infrastructure. We didn't find anything to not like and are currently moving into that direction.

How we do things in Swarm:

1. Monitoring using an updated Swarmprom stack (https://github.com/neuroforgede/swarmsible/tree/master/envir...)

For anyone interested in this, check out: https://github.com/neuroforgede/swarmsible/blob/master/environments/test/test-swarm/stacks/02_monitoring/README.md