osv VS talos

I've been looking at a few.

https://github.com/cloudius-systems/osv

https://ops.city/ (also nanovms) - this is one that I actually got working to at least demo state

I have also found OSv to be interesting.

https://osv.io/

Wow, just now seeing this topic. I work for a cloud company hosted in AWS. We started out, Netflix/Spotify style microservices. We were all on ec2 images generate by packer (and later with AWS Image Factory). When Docker hit, we kicked the tires but never did anything with it beyond using it for running unit tests, and later, infrastructure tests. 5 years ago, during a hackathon, our little group began experimenting with Unikernels, or library operating systems. Interestingly enough, these Unikernels were all stripped down BSD kernels. OSv is FreeBSD based, and Rumprun is NetBSD based. Services running in EC2 on Unikernels would spin up and start sending and receiving traffic before the AWS EC2 healthchecks completed. They are blazing fast! Only problem in 2017, was the tooling. It would have taken too much effort to use Unikernals with our infrastructure. As soon as they start making Unikernels that can run Java bytecode like native code, the fate of containerization will be sealed, IMO. We could get basic JVM webservers running on OSv, but not Cassandra, not Kafka, not yet. OSv now runs on Firecracker, but I have not tried it out, yet. Some links if you are interested: OSv: https://osv.io Rumprun: https://github.com/rumpkernel/rumprun We used this tooling during the Hackathon, but doesn't look like it has been touched in 3 years: https://github.com/solo-io/unik Unikraft Unikernel Dev kit: https://unikraft.org/ And don't forget Firecracker running in Kubernetes https://www.weave.works/oss/firekube/ And of course, being a FreeBSD subreddit, let's not forget FreeBSD on Firecracker https://www.daemonology.net/blog/2022-10-18-FreeBSD-Firecracker.html

Whats the difference to OSv?

https://github.com/cloudius-systems/osv

OSv Unikernel — Optimizing Guest OS to Run Stateless and Serverless Apps in the Cloud — Waldek Kozaczuk [OSv Committer] Unikernels have been demonstrated to deliver excellent performance in terms of throughput and latency, while providing high isolation. However they have also been shown to underperform in some types of workloads when compared to a generic OS like Linux. In this presentation, we demonstrate that certain types of workloads - web servers, microservices, and other stateless and/or serverless apps - can greatly benefit from OSv optimized networking stack and other features. We describe number of experiments where OSv outperforms Linux guest: most notably we note 1.6 throughput (req/s) and 0.6 latency improvement (at p99 percentile) when running nginx and 1.7 throughput (req/s) and 0.6 latency improvement (at p99 percentile) when running simple microservice implemented in Golang. We also show that OSv' small kernel, low boot time and memory consumption allow for very high density when running server-less workloads. The experiment described in this presentation shows we can boot 1,800 OSv microVMs per second on AWS c5n.metal machine with 72 CPUs (25 boots/sec on single CPU) with guest boot time recorded as low as 8.98ms at p50 and 31.49ms at p99 percentile respectively. Lastly we also demonstrate how to automate the build process of the OSv kernel tailored exactly to the specific app and/or VMM so that only the code and symbols needed are part of the kernel and nothing more. OSv is an open source project and can be found at https://github.com/cloudius-systems/osv.

I guess you could have a JVM like that, but not OpenJDK. There is, however, a unikernel that supports running itself and OpenJDK in the same process: http://osv.io/

> Oracle used to offer an installation mode like this

Oracle, and BEA before them, used to offer a JVM which ran on top of a thin custom OS designed only to host the JVM, you could call it a "unikernel". Product was called JRockit Virtual Edition (JRVE), WebLogic Server Virtual Edition (WLS-VE, when used to run WebLogic), earlier BEA called it LiquidVM. The internal name for that thin custom OS was in fact "Bare Metal". Similar in concept to https://github.com/cloudius-systems/osv but completely different implementation

I think one thing which caused a problem for it, is a lot of customers want to deploy various management tools to their VMs (security auditing software, performance monitoring software, etc) and when your VM runs a custom OS that becomes very difficult or impossible. So adopting this product could lead to the pain of having to ask for exceptions to policies requiring those tools and then defending the decision to adopt it against those who use those policies to argue against it. I think this is part of why the product was discontinued.

Nowadays, Oracle offers "bare metal servers" [1] – which are just hypervisor-less servers, same as other cloud vendors do. Or similarly, "Oracle Database Appliance Bare Metal System" [2] – which just means not installing a hypervisor on your Oracle Database Appliance.

So Oracle seems to have a history of using the phrase "bare metal" in both the senses being discussed here.

[1] https://www.oracle.com/cloud/compute/bare-metal.html

[2] https://docs.oracle.com/en/engineered-systems/oracle-databas...

Super cool. I always enjoy reading about systems that challenge, well, "ossified" assumptions. An OS not providing a shell, for example? Madness! ... or is it genius, if the OS has a specific purpose...? It's thought-provoking, if nothing else.

I'm a bit skeptical of parts. For instance, the "init" binary being less than 400 lines of golang - wow! And sure, main.go [1] is less than 400 lines and very readable. Then you squint at the list of imported packages, or look to the left at the directory list and realize main.go isn't nearly the entire init binary.

That `talosctl list` invocation [2] didn't escape my notice either. Sure, the base OS may have only a handful of binaries - how many of those traditional utilities have been stuffed into the API server? Not that I disagree with the approach! I think every company eventually replaces direct shell access with a daemon like this. It's just that "binary footprint" can get a bit funny if you have a really sophisticated API server sitting somewhere.

[1]: https://github.com/siderolabs/talos/blob/main/internal/app/m...

[2]: https://www.talos.dev/v1.6/reference/cli/#talosctl-list

Where `kube.cue` sets reasonable defaults (e.g. image is /). The "cluster" runs on a mini PC in my basement, and I have a small Digital Ocean VM with a static IP acting as an ingress (networking via Tailscale). Backups to cloud storage with restic, alerting/monitoring with Prometheus/Grafana, Caddy/Tailscale for local ingress.

[1] https://www.talos.dev/

[2] https://cuelang.org/

Looks somewhat similar to the talos Linux project[1]

[1] https://www.talos.dev/

Talos Linux basically implements their entire userspace in Go and its similar to BottleRocketOS, because it is designed to host Kubernetes.

https://www.talos.dev/

You might be surprised to find that Talos os (linux distro for kubernetes) mostly uses Go: https://github.com/siderolabs/talos

I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use.

The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environment setups.

I even use it with Longhorn https://longhorn.io/ for shared block storage on the mini cluster.

If anyone uses it with MicroOS, just make sure you switch to kured https://kured.dev/ for the transactional-updates reboot method.

I'd love to compare it against Talos https://www.talos.dev/ but their lack of support for a persistent storage partition (only separate storage device) really hurts most small home / office usage I'd want to try.

If you’re interested in something not AWS check out Talos https://www.talos.dev/

It’s been around longer than Bottlerocket

Can't talk about work, but my homelab is Azure and Oracle managed k8s (AKS/OKE), with onprem Talos soon (Turing Pi 2). My Flux monorepo has the details. OKE performs noticably worse (update cycle, features, control plane performance), but it provides 4 ARM cores and 24GB RAM free so I can't complain

Talos