runc VS virtual-kubelet

I can speak to this. Containers, and by extension k8s, break a well known security boundary that has existed for a very long time - whether you are using a real (hardware) server or a virtual machine on the cloud if you pop that instance/server generally speaking you only have access to that server. Yeh, you might find a db config with connection details if you landed on say a web app host but in general you still have to work to start popping the next N servers.

That's not the case when you are running in k8s and the last container breakout was just announced ~1 month ago: https://github.com/opencontainers/runc/security/advisories/G... .

At the end of the day it is simply not a security boundary. It can solve other problems but not security ones.

It's interesting that, in light of things like this, you still see large software companies adding support for new components written in non-memory safe languages (e.g. C)

As an example Red Hat OpenShift added support for crun(https://github.com/containers/crun) this year(https://cloud.redhat.com/blog/whats-new-in-red-hat-openshift...), which is written in C as an alternative to runc, which is written in Go(https://github.com/opencontainers/runc)...

Rabbit hole indeed. That wasn't related to my job at the time, lol. The job change came with a company-provided computer and that put an end to the tinkering.

BTW, I found my hacks to make runc run on Chromebook: https://github.com/opencontainers/runc/compare/main...gabrys...

being the main author of crun, I can clarify that statement: I am not a fan of Go _for this particular use case_.

Using C instead of Go avoided a bunch of the workarounds that exists in runc to workaround the Go runtime, e.g. https://github.com/opencontainers/runc/blob/main/libcontaine...

runc

Not OP, but if I had to guess, a lot of this can be picked up by just observing common security issues in the Linux space, since similar mistakes and oversights have caused quite a few real-world CVEs in the past, e.g. this random example of a TOCTTOU vulnerability in runc.

Speaking of k8s, anyone here know of ready-made solutions for getting XCode (i.e. xcodebuild) running in pods? As far as I'm aware, there are no good solutions for getting XCode running on Linux, so at the moment I'm just futzing about with a virtual-kubelet[0] implementation that spawns MacOS VMs. This works just fine, but the problem seems like such an obvious one that I expect there to be some existing solution(s) I just missed.

[0]:https://github.com/virtual-kubelet/virtual-kubelet/

Have you looked into virtual kubelet yet? It allows you to make a virtual node in your on-prem cluster that schedules workloads on services like AWS Fargate or Azure Container Instances.

If you are serious about implementing this yourself, you may want to look into virtual kubelet: https://virtual-kubelet.io/

I just stumbled upon this project a while back and don't have experience with it, so I don't know how well it works and what caveats you may face, but there's Virtual Kubelet, which aims to do just that, i.e. running a virtual Kubernetes node outside the cluster. Its Kip provider sounds like the thing you're looking for.

Are you looking something like the following: https://github.com/virtual-kubelet/virtual-kubelet

Kubeedge docs are light on self-justification... How does https://github.com/kubeedge/kubeedge differ from https://github.com/virtual-kubelet/virtual-kubelet or just running a regular kubelet on that edge machine?

If this sounds interesting, do check out Virtual Nodes in Azure Kubernetes Service to see how you can use them to seamlessly scale your applications to Azure Container Instances and benefit from quick provisioning of pods, and only pay per second for their execution time. The virtual nodes add-on for AKS, is based on the open source project Virtual Kubelet which is an open source Kubernetes kubelet implementation.

Use cases like these are made possible by projects like KubeEdge , K3s and Virtual Kubelets. You can read more about how they power the edge with different architectures and compromises here.

This is a complicated question to answer.

This isn't my expertise (the cluster orchestration system), but I can answer to the best of my abilities: Titus, today is a system that sits on top of Kubernetes, and uses Kubernetes components to do its thing, but we've substituted many of the systems with our own. For example, closer to my area of knowledge, we've used our own executor / provider along with the Virtual Kubelet project (https://github.com/virtual-kubelet/virtual-kubelet) instead of Kubelet.

We're exploring where we can leverage the Kubernetes ecosystem, adapt components, or help contribute changes back that others can leverage to enable our use of more COTS components of Kubernetes.

tl;dr: We're swapping out the engines while in flight