gvisor VS sysbox

If you think about it virtualization is just a narrowing of the application-kernel interface. In a standard setting the application has a wide kernel interface available to it with dozens (ex. seccomp) to 100's of syscalls. A vulnerablility in any one of which could result in complete system compromise.

With virtualization the attack surface is narrowed to pretty much just the virtualization interface.

The problem with current virtualization (or more specifically, the VMM's) is that it can be cumbersome, for example memory management is a serious annoyance. The kernel is built to hog memory for cache and etc. but you don't want the guest to be doing that - since you want to overcommit memory as guests will rarely use 100% of what is given to them (especially when the guest is just a jailed singular application), workarounds such as free page reporting and drop_caches hacks exist.

I would expect eventually to see high performance custom kernels for a application jails - for example: gVisor[1] acts as a syscall interceptor (and can use KVM too!) and a custom kernel. Or a modified linux kernel with patched pain points for the guest.

[1] <https://gvisor.dev/>

Hillai: Following our research, Alibaba took several steps to address the vulnerabilities we discovered. They limited image pull secret permissions to read-only access, preventing unauthorized uploads. Additionally, they implemented a secure container technology similar to Google's gVisor project. This technology hardens containers and makes them more difficult to escape from, adding another layer of security.

If you want to use netstack without Bazel, just use the go branch:

https://github.com/google/gvisor/tree/go

go get gvisor.dev/gvisor/pkg/tcpip@go

The go branch is auto generated with all of the generated code checked in.

Isn't gVisor kind of this as well?

"gVisor is an application kernel for containers. It limits the host kernel surface accessible to the application while still giving the application access to all the features it expects. Unlike most kernels, gVisor does not assume or require a fixed set of physical resources; instead, it leverages existing host kernel functionality and runs as a normal process. In other words, gVisor implements Linux by way of Linux."

https://github.com/google/gvisor

This sort of feels like seeing someone riding a bike and saying: why don’t they just get a car? The simple fact is that containers and VMs are quite different. Whether something uses VMX and friends or not is also a red herring, as gVisor also “rolls it own VMM” [1].

[1] https://github.com/google/gvisor/tree/master/pkg/sentry/plat...

Moreover, you might benefit from taking a look at sysbox, a VM-like container runtime that provides a more secure environment. Sysbox is worth it, especially if the main app is running in a container, which means that you'll be running Docker in Docker.

You are probably referring to Sysbox (https://github.com/nestybox/sysbox), which I believe will meet your requirements (systemd, inner containers, security, etc).

Btw, Sysbox is already supported in Docker-Desktop (business tier only), so you can easily do what you want with this instruction:

$ docker run -it --rm -e SYSBOX_SYSCONT_MODE=TRUE nestybox/ubuntu-focal-systemd-docker:latest bash

Disclaimer: I'm Sysbox's co-creator and currently working for Docker.

One project in this space that looked quite promising to me is sysbox[0]. I've used them once for a gitlab runner set-up similar to what is described in their blog[1].

It's currently working great and I have not had any major crashes/incidents for at least the past 8 months.

[0]: https://github.com/nestybox/sysbox

[1]: https://blog.nestybox.com/2020/10/21/gitlab-dind.html

Today, things are very different. Docker-in-Docker has a more secure and safe approach with rootless containers and freemium tools like sysbox. Tools like sysbox let you run Docker-in-Docker without the -privileged flag and optimizes specific scenarios, like running multiple nodes of a Kubernetes cluster as ordinary containers.

Right now I am going with sysbox rootless containers. https://github.com/nestybox/sysbox

We’ve been using Sysbox (https://github.com/nestybox/sysbox) for our Buildkite based CI/CD setup, allows docker-in-docker without privileged containers. Paired with careful IAM/STS design we’ve ended up with isolated job containers with their own IAM roles limited to least-privilege.

A good alternative to the VM approach is to use Kubernetes + Sysbox (a next-gen "runc", free, open-source).