design VS documentation

> we'd have been fighting cni complexity to make it work.

Appreciate the candid responses, thanks for taking the time. That ipv6 wireguard peering post was really fascinating I read that too. Wireguard has been quite the a game-changer in it's space as well and a lot of value IMO is just in the simplicity and difficulty of misconfiguration, even though the performance is also fantastic.

Grateful that ya'll are sharing what you're doing right/finding interesting.

Since ya'll might appreciate this, I think there's an ultimate form of all these orchestrators out there that boils everything down to the "operator pattern" -- I call it "buhzaar" but I tried to get my thoughts out of the notebook a while ago[0]. It's almost like a completely normalized DB might be -- to strip an orchestrator down to it's bare minimum, which facilitates other processes that do resource provisioning and management. Then let people bring their own things that provision resources (and maybe you some "officially supported" ones but they all live separately and iterate separately).

I didn't quite put down all the thoughts I had but you think this is too much normalization (in the same way no one wants to do 7 joins)? You could argue that both nomad and k8s are denormalized (they intrinsically "know" how to provision/manage certain things) to a certain extent, and nomad just "bundles" less.

[0]: https://gitlab.com/buhzaar/design

Please feel free too, would love to chat about this. I think we think extremely similarly -- What you're trying to build is almost exactly what I'm trying to build, except I plan on getting my leverage from k8s (and eventually my own thing that I'm working on called buhzaar which aims to be simpler than k8s).

Both Kata Containers and UTM support virtio-fs, so this is not strictly true. The former can be used as a stand-in replacement for the runtime used by docker desktop[1]. With the latter, one could use a UTM-backed guest as a docker runtime in macOS[2] or run docker directly on the guest[3].

[1] https://github.com/kata-containers/documentation/blob/master...

[2] https://www.codeluge.com/post/setting-up-docker-on-macos-m1-...

[3] https://www.lifeintech.com/2021/11/03/docker-performance-on-...

For services with increased security requirements, it is recommended to use a low-level run-time with a high degree of isolation (gVisior, Kata-runtime)

On the official Kata repo, I found a tutorial only for manually deployed Kubernetes on GCE.

This is new and may not be used much, but it is possible to use part of Kata with part of Firecracker. https://github.com/kata-containers/documentation/wiki/Initia...

If it's using firecracker, it's probably using KVM virtualization while ensuring that the memory the VM consumes is not pinned... that is, that the VM can be swapped out of memory. For reference, firecracker was created by AWS to run and secure AWS Lambda. The hypervisor is written in rust and uses seccomp to eliminate unnecessary system calls. They open sourced it a few years back.

What you gain is a stronger security boundary. Just FYI, since 2019, you can also do this in Kubernetes using Kata containers which will happily shim firecracker. The setup is not simple though.

https://github.com/kata-containers/documentation/wiki/Initia...

Overall, fly.io building infrastructure on this pattern is fantastic and making it accessible is fantastic. Looking forward to seeing how this continues to evolve and am happy to see more infra build on top of firecracker. Very exciting!