firecracker-container VS documentation

There is this project, which I have never used, but seems promising. https://github.com/firecracker-microvm/firecracker-container...

How does that compare to firecracker-containerd?

https://github.com/firecracker-microvm/firecracker-container...

This repository enables the use of a container runtime, containerd, to manage Firecracker microVMs. Like traditional containers, Firecracker microVMs offer fast start-up and shut-down and minimal overhead. Unlike traditional containers, however, they can provide an additional layer of isolation via the KVM hypervisor.

I'm really impressed by fly.io, and the candidness with which they share some of their really awesome technology. Being container-first is the next step for PaaS IMO and they are ahead of the pack.

I aim to build a platform like theirs someday (probably not any time soon) but I don't think I'd do any of what they're doing -- it feels unnecessary. Bear with me as I recently learned that they use nomad[0] and some of these suggestions are kubernetes projects but I'd love to hear why the following technologies were decided against (if they were):

- kata-containers[1] (it does the whole container -> VM flow for you, automatically, nemu, firecracker) with multiple VMM options[2]

- linuxkit[3] (let's say you didn't go with kata-containers, this is another container->VM path)

- firecracker-containerd[4] (very minimal keep-your-container-but-run-it-as-a-VM)

- kubevirt[5] (if you just want to actually run VMs, regardless of how you built them)

- Ceph[6] for storage -- make LVM pools and just give them to Ceph, you'll get blocks, distributed filesystems (CephFS), and object gateways (S3/Swift) out of it (in the k8s space Rook manages this)

As an aside to all this, there's also LXD, which supports running "system" (user namespace isolated) containers, VMs (somewhat recent[7][8]), live migration via criu[9], management/migration of underlying filesystems, runs on LVM or zfs[10], it's basically all-in-one, but does fall behind in terms of ecosystem since everyone else is aboard the "cloud native"/"works-with-kubernetes" train.

I've basically how I plan to run a service like fly.io if I ever did -- so maybe my secret is out, but I sure would like to know just how much of this fly.io got built on (if any of it), and/or what was turned down.

[0]: https://news.ycombinator.com/item?id=26745514

[1]: https://github.com/kata-containers/kata-containers

[2]: https://github.com/kata-containers/kata-containers/blob/2fc7...

[3]: https://github.com/linuxkit/linuxkit

[4]: https://github.com/firecracker-microvm/firecracker-container...

[5]: https://github.com/kubevirt/kubevirt

[6]: https://docs.ceph.com/

[7]: https://discuss.linuxcontainers.org/t/running-virtual-machin...

[8]: https://github.com/lxc/lxd/issues/6205

[9]: https://criu.org/Main_Page

[10]: https://linuxcontainers.org/lxd/docs/master/storage

Both Kata Containers and UTM support virtio-fs, so this is not strictly true. The former can be used as a stand-in replacement for the runtime used by docker desktop[1]. With the latter, one could use a UTM-backed guest as a docker runtime in macOS[2] or run docker directly on the guest[3].

[1] https://github.com/kata-containers/documentation/blob/master...

[2] https://www.codeluge.com/post/setting-up-docker-on-macos-m1-...

[3] https://www.lifeintech.com/2021/11/03/docker-performance-on-...

For services with increased security requirements, it is recommended to use a low-level run-time with a high degree of isolation (gVisior, Kata-runtime)

On the official Kata repo, I found a tutorial only for manually deployed Kubernetes on GCE.

This is new and may not be used much, but it is possible to use part of Kata with part of Firecracker. https://github.com/kata-containers/documentation/wiki/Initia...

If it's using firecracker, it's probably using KVM virtualization while ensuring that the memory the VM consumes is not pinned... that is, that the VM can be swapped out of memory. For reference, firecracker was created by AWS to run and secure AWS Lambda. The hypervisor is written in rust and uses seccomp to eliminate unnecessary system calls. They open sourced it a few years back.

What you gain is a stronger security boundary. Just FYI, since 2019, you can also do this in Kubernetes using Kata containers which will happily shim firecracker. The setup is not simple though.

https://github.com/kata-containers/documentation/wiki/Initia...

Overall, fly.io building infrastructure on this pattern is fantastic and making it accessible is fantastic. Looking forward to seeing how this continues to evolve and am happy to see more infra build on top of firecracker. Very exciting!