kata-containers VS sysbox

> I actually wonder how much "overhead" a VM actually has. i.e. a linux kernel that doesn't do anything (say perhaps just boots to an init that mounts proc and every n seconds read in/prints out /proc/meminfo) how much memory would the kernel actually be using?

There's already some memory sharing available using DAX in Kata Containers at least: https://github.com/kata-containers/kata-containers/blob/main...

Seems like Fly.io Machines are trying reimplement Kata Containers with the Firecracker backend [0].

Kata has a guest image and guest agent to run multiple isolated containers [1].

[0] https://katacontainers.io/

[1] https://github.com/kata-containers/kata-containers/blob/main...

> Last time I looked (a few months ago), the documentation was pretty sparse or outdated.

It still is, though it works somewhat seamlessly when installing with https://github.com/kata-containers/kata-containers/blob/main...

Though only one of the hypervisors works well.

Better to use a secure VM, can even get container-like VMs with kata-containers

As I understand,Kata Containers

Kubernetes using Kata containers as a containerd backend

https://github.com/kata-containers/kata-containers/blob/main...

My home k8s cluster is now "locked down" using micro-vms (kata-containers[0]), pod level firewalling (cilium[1]), permission-limited container users, and mostly immutable environments. Given how quickly I rolled this out; the tools to enhance cluster environment security seem more accessible now than my previous research a few years ago.

I know it's not exactly a production setup, but I really do feel that it's the most secure runtime environment I've ever had accessible at home. Probably more so than my desktops, which you could argue undermines most of my effort, but I like to think I'm pretty careful.

In the beginning I was very skeptical, but being able to just build a docker/OCI image and then manage its relationships with other services with "one pane of glass" that I can commit to git is so much simpler to me than my previous workflows. My previous setup involved messing with a bunch of tools like packer, cloud-init, terraform, ansible, libvirt, whatever firewall frontend was on the OS, and occasionally sshing in for anything not covered.

[0] https://github.com/kata-containers/kata-containers

Moreover, you might benefit from taking a look at sysbox, a VM-like container runtime that provides a more secure environment. Sysbox is worth it, especially if the main app is running in a container, which means that you'll be running Docker in Docker.

You are probably referring to Sysbox (https://github.com/nestybox/sysbox), which I believe will meet your requirements (systemd, inner containers, security, etc).

Btw, Sysbox is already supported in Docker-Desktop (business tier only), so you can easily do what you want with this instruction:

$ docker run -it --rm -e SYSBOX_SYSCONT_MODE=TRUE nestybox/ubuntu-focal-systemd-docker:latest bash

Disclaimer: I'm Sysbox's co-creator and currently working for Docker.

One project in this space that looked quite promising to me is sysbox[0]. I've used them once for a gitlab runner set-up similar to what is described in their blog[1].

It's currently working great and I have not had any major crashes/incidents for at least the past 8 months.

[0]: https://github.com/nestybox/sysbox

[1]: https://blog.nestybox.com/2020/10/21/gitlab-dind.html

Today, things are very different. Docker-in-Docker has a more secure and safe approach with rootless containers and freemium tools like sysbox. Tools like sysbox let you run Docker-in-Docker without the -privileged flag and optimizes specific scenarios, like running multiple nodes of a Kubernetes cluster as ordinary containers.

Right now I am going with sysbox rootless containers. https://github.com/nestybox/sysbox

We’ve been using Sysbox (https://github.com/nestybox/sysbox) for our Buildkite based CI/CD setup, allows docker-in-docker without privileged containers. Paired with careful IAM/STS design we’ve ended up with isolated job containers with their own IAM roles limited to least-privilege.

A good alternative to the VM approach is to use Kubernetes + Sysbox (a next-gen "runc", free, open-source).