talos VS minikube

Super cool. I always enjoy reading about systems that challenge, well, "ossified" assumptions. An OS not providing a shell, for example? Madness! ... or is it genius, if the OS has a specific purpose...? It's thought-provoking, if nothing else.

I'm a bit skeptical of parts. For instance, the "init" binary being less than 400 lines of golang - wow! And sure, main.go [1] is less than 400 lines and very readable. Then you squint at the list of imported packages, or look to the left at the directory list and realize main.go isn't nearly the entire init binary.

That `talosctl list` invocation [2] didn't escape my notice either. Sure, the base OS may have only a handful of binaries - how many of those traditional utilities have been stuffed into the API server? Not that I disagree with the approach! I think every company eventually replaces direct shell access with a daemon like this. It's just that "binary footprint" can get a bit funny if you have a really sophisticated API server sitting somewhere.

[1]: https://github.com/siderolabs/talos/blob/main/internal/app/m...

[2]: https://www.talos.dev/v1.6/reference/cli/#talosctl-list

Where `kube.cue` sets reasonable defaults (e.g. image is /). The "cluster" runs on a mini PC in my basement, and I have a small Digital Ocean VM with a static IP acting as an ingress (networking via Tailscale). Backups to cloud storage with restic, alerting/monitoring with Prometheus/Grafana, Caddy/Tailscale for local ingress.

[1] https://www.talos.dev/

[2] https://cuelang.org/

Looks somewhat similar to the talos Linux project[1]

[1] https://www.talos.dev/

Talos Linux basically implements their entire userspace in Go and its similar to BottleRocketOS, because it is designed to host Kubernetes.

https://www.talos.dev/

You might be surprised to find that Talos os (linux distro for kubernetes) mostly uses Go: https://github.com/siderolabs/talos

I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use.

The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environment setups.

I even use it with Longhorn https://longhorn.io/ for shared block storage on the mini cluster.

If anyone uses it with MicroOS, just make sure you switch to kured https://kured.dev/ for the transactional-updates reboot method.

I'd love to compare it against Talos https://www.talos.dev/ but their lack of support for a persistent storage partition (only separate storage device) really hurts most small home / office usage I'd want to try.

If you’re interested in something not AWS check out Talos https://www.talos.dev/

It’s been around longer than Bottlerocket

Can't talk about work, but my homelab is Azure and Oracle managed k8s (AKS/OKE), with onprem Talos soon (Turing Pi 2). My Flux monorepo has the details. OKE performs noticably worse (update cycle, features, control plane performance), but it provides 4 ARM cores and 24GB RAM free so I can't complain

Talos

With the containerized Node.js/Express API, I could run multiple containers, scaling to handle more traffic. Using a tool called minikube, we can easily spin up a local Kubernetes cluster to horizontally scale Docker containers. It was possible to keep one shared instance of the database, and many APIs were routed with an internal Kubernetes load balancer.

Install Minicube - a tool that allows us to spin up a Kubernetes cluster in a local machine Run minikube start to start your Kubernetes cluster Run minikube dashboard to spin up a web-based user interface that allows you to manage your Kubernetes cluster

If you're just messing around, just use kind (https://kind.sigs.k8s.io) or minikube if you want VMs (https://minikube.sigs.k8s.io). Both work on ARM-based platforms.

You can also use k3s; it's hella easy to get started with and it works great.

$ minikube addons enable dashboard 💡 dashboard is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub. You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS ▪ Using image docker.io/kubernetesui/dashboard:v2.7.0 ▪ Using image docker.io/kubernetesui/metrics-scraper:v1.0.8 🌟 The 'dashboard' addon is enabled $ minikube addons enable metrics-server 💡 metrics-server is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub. You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS ▪ Using image registry.k8s.io/metrics-server/metrics-server:v0.6.4 🌟 The 'metrics-server' addon is enabled $ minikube addons enable ingress 💡 ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub. You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS 💡 After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1" ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407 ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.8.1 ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407 🔎 Verifying ingress addon... 🌟 The 'ingress' addon is enabled

A Kubernetes distribution: You need to install a Kubernetes distribution to create the Kubernetes cluster and other necessary resources, such as deployments and services. This tutorial uses kind (v0.18.0), but you can use any other Kubernetes distribution, including minikube or K3s.

3.Minikube ( https://minikube.sigs.k8s.io ):

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES7523fd2c20c7 gcr.io/google\_containers/k8s-dns-sidecar-amd64 "/sidecar --v=2 --..." 18 hours ago Up 18 hours k8s\_sidecar\_kube-dns-86f6f55dd5-qwc6z\_kube-system\_c1333ffc-e4d6-11e7-bccf-0021ccbf0996\_09bd438011406 gcr.io/google\_containers/k8s-dns-dnsmasq-nanny-amd64 "/dnsmasq-nanny -v..." 18 hours ago Up 18 hours k8s\_dnsmasq\_kube-dns-86f6f55dd5-qwc6z\_kube-system\_c1333ffc-e4d6-11e7-bccf-0021ccbf0996\_05c35e00a5a27 gcr.io/google\_containers/k8s-dns-kube-dns-amd64 "/kube-dns --domai..." 18 hours ago Up 18 hours k8s\_kubedns\_kube-dns-86f6f55dd5-qwc6z\_kube-system\_c1333ffc-e4d6-11e7-bccf-0021ccbf0996\_077ef463642b7 gcr.io/google\_containers/pause-amd64:3.0 "/pause" 18 hours ago Up 18 hours k8s\_POD\_kube-dns-86f6f55dd5-qwc6z\_kube-system\_c1333ffc-e4d6-11e7-bccf-0021ccbf0996\_039f618666205 gcr.io/google\_containers/kubernetes-dashboard-amd64 "/dashboard --inse..." 18 hours ago Up 18 hours k8s\_kubernetes-dashboard\_kubernetes-dashboard-vgpjl\_kube-system\_c1176a44-e4d6-11e7-bccf-0021ccbf0996\_0023b7b554a8c gcr.io/google\_containers/pause-amd64:3.0 "/pause" 18 hours ago Up 18 hours k8s\_POD\_kubernetes-dashboard-vgpjl\_kube-system\_c1176a44-e4d6-11e7-bccf-0021ccbf0996\_01c3bdb7bdeb1 gcr.io/google-containers/kube-addon-manager "/opt/kube-addons.sh" 18 hours ago Up 18 hours k8s\_kube-addon-manager\_kube-addon-manager-tpad\_kube-system\_7b19c3ba446df5355649563d32723e4f\_08a00feefa754 gcr.io/google\_containers/pause-amd64:3.0 "/pause" 18 hours ago Up 18 hours k8s\_POD\_kube-addon-manager-tpad\_kube-system\_7b19c3ba446df5355649563d32723e4f\_0b657eab5f6f5 gcr.io/k8s-minikube/storage-provisioner "/storage-provisioner" 18 hours ago Up 18 hours k8s\_storage-provisioner\_storage-provisioner\_kube-system\_c0a8b187-e4d6-11e7-bccf-0021ccbf0996\_067be5cc1dd0d gcr.io/google\_containers/pause-amd64:3.0 "/pause" 18 hours ago Up 18 hours k8s\_POD\_storage-provisioner\_kube-system\_c0a8b187-e4d6-11e7-bccf-0021ccbf0996\_0 I just did the Kubernetes minikube tutorial at https://github.com/kubernetes/minikube, and I cannot stop or remove these containers, they always get recreated.

https://github.com/kubernetes/minikube for local learning that's lightweight.

Haven’t dabbled with minikube yet, but there’s a whole thread about this error here: https://github.com/kubernetes/minikube/issues/9798