kata-containers VS lxd

Seems like Fly.io Machines are trying reimplement Kata Containers with the Firecracker backend [0].

Kata has a guest image and guest agent to run multiple isolated containers [1].

[0] https://katacontainers.io/

[1] https://github.com/kata-containers/kata-containers/blob/main...

> Last time I looked (a few months ago), the documentation was pretty sparse or outdated.

It still is, though it works somewhat seamlessly when installing with https://github.com/kata-containers/kata-containers/blob/main...

Though only one of the hypervisors works well.

Better to use a secure VM, can even get container-like VMs with kata-containers

As I understand,Kata Containers

Kubernetes using Kata containers as a containerd backend

https://github.com/kata-containers/kata-containers/blob/main...

My home k8s cluster is now "locked down" using micro-vms (kata-containers[0]), pod level firewalling (cilium[1]), permission-limited container users, and mostly immutable environments. Given how quickly I rolled this out; the tools to enhance cluster environment security seem more accessible now than my previous research a few years ago.

I know it's not exactly a production setup, but I really do feel that it's the most secure runtime environment I've ever had accessible at home. Probably more so than my desktops, which you could argue undermines most of my effort, but I like to think I'm pretty careful.

In the beginning I was very skeptical, but being able to just build a docker/OCI image and then manage its relationships with other services with "one pane of glass" that I can commit to git is so much simpler to me than my previous workflows. My previous setup involved messing with a bunch of tools like packer, cloud-init, terraform, ansible, libvirt, whatever firewall frontend was on the OS, and occasionally sshing in for anything not covered.

[0] https://github.com/kata-containers/kata-containers

I'm really impressed by fly.io, and the candidness with which they share some of their really awesome technology. Being container-first is the next step for PaaS IMO and they are ahead of the pack.

I aim to build a platform like theirs someday (probably not any time soon) but I don't think I'd do any of what they're doing -- it feels unnecessary. Bear with me as I recently learned that they use nomad[0] and some of these suggestions are kubernetes projects but I'd love to hear why the following technologies were decided against (if they were):

- kata-containers[1] (it does the whole container -> VM flow for you, automatically, nemu, firecracker) with multiple VMM options[2]

- linuxkit[3] (let's say you didn't go with kata-containers, this is another container->VM path)

- firecracker-containerd[4] (very minimal keep-your-container-but-run-it-as-a-VM)

- kubevirt[5] (if you just want to actually run VMs, regardless of how you built them)

- Ceph[6] for storage -- make LVM pools and just give them to Ceph, you'll get blocks, distributed filesystems (CephFS), and object gateways (S3/Swift) out of it (in the k8s space Rook manages this)

As an aside to all this, there's also LXD, which supports running "system" (user namespace isolated) containers, VMs (somewhat recent[7][8]), live migration via criu[9], management/migration of underlying filesystems, runs on LVM or zfs[10], it's basically all-in-one, but does fall behind in terms of ecosystem since everyone else is aboard the "cloud native"/"works-with-kubernetes" train.

I've basically how I plan to run a service like fly.io if I ever did -- so maybe my secret is out, but I sure would like to know just how much of this fly.io got built on (if any of it), and/or what was turned down.

[0]: https://news.ycombinator.com/item?id=26745514

[1]: https://github.com/kata-containers/kata-containers

[2]: https://github.com/kata-containers/kata-containers/blob/2fc7...

[3]: https://github.com/linuxkit/linuxkit

[4]: https://github.com/firecracker-microvm/firecracker-container...

[5]: https://github.com/kubevirt/kubevirt

[6]: https://docs.ceph.com/

[7]: https://discuss.linuxcontainers.org/t/running-virtual-machin...

[8]: https://github.com/lxc/lxd/issues/6205

[9]: https://criu.org/Main_Page

[10]: https://linuxcontainers.org/lxd/docs/master/storage

Kata Containers https://github.com/kata-containers/kata-containers

Please correct me if I'm wrong, but the post also links the "add Canonical CLA check #12665" [0], and my understanding is that "retain copyright" here is like a typical forum agreement where you going forward must agree to a perpetual worldwide unlimited license to Canonical that they can use as they please per [1]:

>In effect, you’re giving us a licence, but you still own the copyright — so you retain the right to modify your code and use it in other projects.

You explicitly do retain ownership, so you can then take that same code and contribute it elsewhere under any license you wish. The same author could contribute the same patch to both the LXD and the Incus fork. But some might object to being required to allow Canonical to specially license as they want.

So your characterization seems unfair, and then gets kind of nasty at the end:

>The author is pissed off because he can't build custom versions without redistributing the modifications

Incus is a full fork, and Canonical has apparently been taking changes back from it as well as is often the case with such forks where both sides get value from each other. It's perfectly understandable for some folks to be bummed if that's no longer the case, and there is nothing evil about the Apache2 license. There's plenty of history that in OSS going back to the beginning, no need for insinuations or attacks.

----

0: https://github.com/canonical/lxd/pull/12665/commits/eb5c773d...

1: https://ubuntu.com/legal/contributors

You could consider LXD which lets you easily run both containers and VMs: https://ubuntu.com/lxd

I hope this doesn't affect LXC negatively.

LXC and LXD share plenty of contributors.

https://github.com/lxc/lxc/graphs/contributors

https://github.com/canonical/lxd/graphs/contributors

I use an "unprivileged LXC container" setup on several Debian bullseye hosts. It works fantastic, and each LXC container feels like a real server.

Compare that to Docker's "one-container-one-process" philosophy, reinventing the wheel by awkwardly composing multiple containers.

[1] https://linuxcontainers.org/lxd/

The expected changes are: - https://github.com/lxc/lxd will now become https://github.com/canonical/lxd - https://linuxcontainers.org/lxd will disappear and be replaced with a mention directing users to https://ubuntu.com/lxd - The LXD YouTube channel will be handed over to the Canonical team - The LXD section on the LinuxContainers community forum will slowly be sunset in favor of the Ubuntu Discourse forum run by Canonical - The LXD CI infrastructure will be moved under Canonical’s care - Image building for Linux Containers will no longer be relying on systems provided by Canonical, limiting image building to x86_64 and aarch64.