design VS linuxkit

> we'd have been fighting cni complexity to make it work.

Appreciate the candid responses, thanks for taking the time. That ipv6 wireguard peering post was really fascinating I read that too. Wireguard has been quite the a game-changer in it's space as well and a lot of value IMO is just in the simplicity and difficulty of misconfiguration, even though the performance is also fantastic.

Grateful that ya'll are sharing what you're doing right/finding interesting.

Since ya'll might appreciate this, I think there's an ultimate form of all these orchestrators out there that boils everything down to the "operator pattern" -- I call it "buhzaar" but I tried to get my thoughts out of the notebook a while ago[0]. It's almost like a completely normalized DB might be -- to strip an orchestrator down to it's bare minimum, which facilitates other processes that do resource provisioning and management. Then let people bring their own things that provision resources (and maybe you some "officially supported" ones but they all live separately and iterate separately).

I didn't quite put down all the thoughts I had but you think this is too much normalization (in the same way no one wants to do 7 joins)? You could argue that both nomad and k8s are denormalized (they intrinsically "know" how to provision/manage certain things) to a certain extent, and nomad just "bundles" less.

[0]: https://gitlab.com/buhzaar/design

Please feel free too, would love to chat about this. I think we think extremely similarly -- What you're trying to build is almost exactly what I'm trying to build, except I plan on getting my leverage from k8s (and eventually my own thing that I'm working on called buhzaar which aims to be simpler than k8s).

Another project that aims to deliver this is Linuxkit (https://github.com/linuxkit/linuxkit). All the components they ship are written in memory safe languages (usually Go) and run as containers under containerd. You can build a custom image very easily, fully defined as a YAML file.

docker-the-company maintained https://github.com/linuxkit/linuxkit when I worked there. I have no idea who maintains it now, but it looks like it is still active (presumably still docker-the-company, since their adopters list [1] lists docker desktop).

[1]: https://github.com/linuxkit/linuxkit/blob/master/ADOPTERS.md

LF-Edge EVE project leverages Linuxkit to create custom OSs for Edge Devices which in turn leverages Containers as Lego Blocks

LinuxKit - Docker

Behind the scenes Docker Desktop for Mac spawns a linuxkit VM with a bit of extra stuff like NFS to enable mounting Mac paths into containers. In the Docker Desktop settings you'll find the current resource assignment for that VM. That is pretty much reserved for docker so that it does not have to compete with MacOS processes for available resources.

That said... you might want to check out linuxkit

The definition of what a unikernel is needs to be narrowed down, a lot of these projects in the space (not all the ones listed above) have material differences that are not clear:

- some run only one language

- some require recompilation

- some essentially swap out libraries, others do something closer to dropping your already mostly static binary in a minimal disk image

- some build pid1 processes, others VMs images

Anyway, here are some additional entries in the space:

- https://ssrg-vt.github.io/hermitux/

- https://github.com/linuxkit/linuxkit (more embedded/minimal VM than unikernel)

- https://nabla-containers.github.io/ (runs on Solo5)

I am going through using Linuxkit to build AMIs for cloud providers now. I wouldn’t necessarily class linuxkit as a universal project because it doesn’t have the hallmark blurring of user and kernel space or kernel-as-a-library but you can customize the kernel so it’s an adjacent idea, and I think it’s the one most likely to be in actual use at non-hyperscalers.

I believe there is growing interest in providing leaner, "trimmed" runtimes for services deployed to the cloud. Today, this is seen largely by specializing the Linux kernel for, for example, container services[0] or in general[1], as much as that is possible (the paper above covers this problem in greater detail). But, Unikernels in themselves are not yet widely adopted. This is the space Unikraft is aiming to enter, providing the ultimate level of specialization for a target application.

It's clear that bigger players, such as Red Hat[2] are interested in the topic of unikernels, and that cloud providers are preparing for this future too [3].

[0]: https://github.com/linuxkit/linuxkit