kata-containers VS cloud-hypervisor

Seems like Fly.io Machines are trying reimplement Kata Containers with the Firecracker backend [0].

Kata has a guest image and guest agent to run multiple isolated containers [1].

[0] https://katacontainers.io/

[1] https://github.com/kata-containers/kata-containers/blob/main...

If you install kata with https://github.com/kata-containers/kata-containers/blob/main...

Then you just use it with:

docker run --runtime io.containerd.kata.v2 --rm -it hello-world

Kara used “DAX” to directly share access of the host filesystem to the guest kernel. I thought this was pretty interesting, but it sounds like a possible spot to start a jailbreak.

> Mapping as a direct access device allows the guest to directly access the host memory pages (such as via Execute In Place (XIP)), bypassing the guest kernel's page cache. This zero copy provides both time and space optimizations.

> Mapping as a direct access device inside the VM allows pages from the host to be demand loaded using page faults, rather than having to make requests via a virtualized device (causing expensive VM exits/hypercalls), thus providing a speed optimization.

> Utilizing mmap(2)'s MAP_SHARED shared memory option on the host allows the host to efficiently share pages.

From https://github.com/kata-containers/kata-containers/tree/main...

> Last time I looked (a few months ago), the documentation was pretty sparse or outdated.

It still is, though it works somewhat seamlessly when installing with https://github.com/kata-containers/kata-containers/blob/main...

Though only one of the hypervisors works well.

As I understand,Kata Containers

Kubernetes using Kata containers as a containerd backend

https://github.com/kata-containers/kata-containers/blob/main...

My home k8s cluster is now "locked down" using micro-vms (kata-containers[0]), pod level firewalling (cilium[1]), permission-limited container users, and mostly immutable environments. Given how quickly I rolled this out; the tools to enhance cluster environment security seem more accessible now than my previous research a few years ago.

I know it's not exactly a production setup, but I really do feel that it's the most secure runtime environment I've ever had accessible at home. Probably more so than my desktops, which you could argue undermines most of my effort, but I like to think I'm pretty careful.

In the beginning I was very skeptical, but being able to just build a docker/OCI image and then manage its relationships with other services with "one pane of glass" that I can commit to git is so much simpler to me than my previous workflows. My previous setup involved messing with a bunch of tools like packer, cloud-init, terraform, ansible, libvirt, whatever firewall frontend was on the OS, and occasionally sshing in for anything not covered.

[0] https://github.com/kata-containers/kata-containers

I'm really impressed by fly.io, and the candidness with which they share some of their really awesome technology. Being container-first is the next step for PaaS IMO and they are ahead of the pack.

I aim to build a platform like theirs someday (probably not any time soon) but I don't think I'd do any of what they're doing -- it feels unnecessary. Bear with me as I recently learned that they use nomad[0] and some of these suggestions are kubernetes projects but I'd love to hear why the following technologies were decided against (if they were):

- kata-containers[1] (it does the whole container -> VM flow for you, automatically, nemu, firecracker) with multiple VMM options[2]

- linuxkit[3] (let's say you didn't go with kata-containers, this is another container->VM path)

- firecracker-containerd[4] (very minimal keep-your-container-but-run-it-as-a-VM)

- kubevirt[5] (if you just want to actually run VMs, regardless of how you built them)

- Ceph[6] for storage -- make LVM pools and just give them to Ceph, you'll get blocks, distributed filesystems (CephFS), and object gateways (S3/Swift) out of it (in the k8s space Rook manages this)

As an aside to all this, there's also LXD, which supports running "system" (user namespace isolated) containers, VMs (somewhat recent[7][8]), live migration via criu[9], management/migration of underlying filesystems, runs on LVM or zfs[10], it's basically all-in-one, but does fall behind in terms of ecosystem since everyone else is aboard the "cloud native"/"works-with-kubernetes" train.

I've basically how I plan to run a service like fly.io if I ever did -- so maybe my secret is out, but I sure would like to know just how much of this fly.io got built on (if any of it), and/or what was turned down.

[0]: https://news.ycombinator.com/item?id=26745514

[1]: https://github.com/kata-containers/kata-containers

[2]: https://github.com/kata-containers/kata-containers/blob/2fc7...

[3]: https://github.com/linuxkit/linuxkit

[4]: https://github.com/firecracker-microvm/firecracker-container...

[5]: https://github.com/kubevirt/kubevirt

[6]: https://docs.ceph.com/

[7]: https://discuss.linuxcontainers.org/t/running-virtual-machin...

[8]: https://github.com/lxc/lxd/issues/6205

[9]: https://criu.org/Main_Page

[10]: https://linuxcontainers.org/lxd/docs/master/storage

Kata Containers https://github.com/kata-containers/kata-containers

There is no mention of cloud-hypervisor[1] (also in the rust-vmm ecosystem) in the article. It has the memory reclamation feature they require. It also support VFIO and virtiofs.

[1] <https://github.com/cloud-hypervisor/cloud-hypervisor>

https://github.com/tandasat/Hypervisor-101-in-Rust is there to help

https://github.com/cloud-hypervisor/cloud-hypervisor isn't educational necessarily but is one of the most technically progressive fastest developing highest funded vm projects ever, and there are oodles of tech talks on it. I am not qualified to make any specific recommendations, but there's tons of stuff here.

> The goal of the Cloud Hypervisor project differs from the aforementioned projects in that it aims to be a general purpose VMM for Cloud Workloads and not limited to container/serverless or client workloads.

Firecracker is such a great piece of technology. I'm amazed that AWS actually open-sourced it. All kudos to them. We're using Firecracker at our company to allow API companies build interactive demos like this one we built for Prisma [1].

[0] https://github.com/cloud-hypervisor/cloud-hypervisor

[1] https://playground.prisma.io

The Chrome OS hypervisor was then evolved/forked into Firecracker and Intel's Cloud Hypervisor, with the latter supporting both Linux and Windows. Perhaps Cloud Hypervisor would serve as a good backbone for sandboxing, with its Rust implementation and focus on security?

GitHub - cloud-hypervisor/cloud-hypervisor: A Virtual Machine Monitor for modern Cloud workloads. Features include CPU, memory and device hotplug, support for running Windows and Linux guests, device offload with vhost-user and a minimal compact footprint. Written in Rust with a strong focus on security.

Did you guys think about live migrations? https://github.com/cloud-hypervisor/cloud-hypervisor seems to support it and it shares a good amount of code with firecracker.

Virtink is a Kubernetes add-on for running Cloud Hypervisor virtual machines. By using Cloud Hypervisor as the underlying hypervisor, Virtink enables a lightweight and secure way to run fully virtualized workloads in a canonical Kubernetes cluster.

A taxing overhead would probably arise if a user relies on a complex file-system inside a VM (for instance, ZFS on the host, and BTRFS inside a VM, which is the default for Fedora installation now). The general idea is to keep the top layer as clean and simple as possible, and to not only make a few clear assumptions about the default kind of virtual machine model Phyllome OS will support : as of now, I have settled on a Q35 chipset with EFI (but the virt chipset, from the Cloud Hypervisor project, will ideally replace the Q35 chipset), virtio-devices (and their vhost counterparts, which are running on their own, outside the virtual machine monitor, and are more performant and secured as far as I understand), VFIO, etc.

Inside the cluster my containers are Linux only. I don't believe kata-containers supports Windows containers as I don't think rust-vmm, which is used by CloudHypervisor[0], or the kata internal execution agent support it.

If I wanted to run Windows in the cluster I'd probably have to look at KubeVirt[1]. KubeVirt is oriented towards getting traditional VM workloads (ones you'd run in QEMU, Hyper-V, etc) functioning in a Kubernetes environment. While kata-containers is oriented towards giving container runtime (docker, containers, CRI-O) based workloads the protection of virtualization, with minimal friction.

Previously external to the cluster I had some Windows VMs hosted on QEMU/KVM + libvirt for experimentation with Linux and Active Directory integration, but they've since been deleted. I've got one OpenBSD server for serving up update images to my routers.

For network infra I have a number of VyOS[2] firewalls both at the edge and between VLANs, and Mikrotik devices for switching.

[0] https://github.com/cloud-hypervisor/cloud-hypervisor

[1] https://github.com/kubevirt/kubevirt

[2] https://www.vyos.io