humility VS vector

A lot of questions in there! Taking these in order:

1. We aren't making standalone servers: the Oxide compute sled comes in the Oxide rack. So are not (and do not intend to be) a drop in replacement for extant rack mounted servers.

2. We have taken a fundamentally different approach to firmware, with a true root of trust that can attest to the service processor -- which can turn attest to the system software. This prompts a lot of questions (e.g., who attests to the root of trust?), and there is a LOT to say about this; look for us to talk a lot more about this

3. In stark contrast (sadly) to nearly everyone else in the server space, the firmware we are developing is entirely open source. More details on that can be found in Cliff Biffle's 2021 OSFC talk and the Hubris and Humility repos.[0][1][2]

4. Definitely not vaporware! We are in the process of shipping to our first customers; you can follow our progress in our Oxide and Friends podcast.[3]

[0] https://www.osfc.io/2021/talks/on-hubris-and-humility-develo...

[1] https://github.com/oxidecomputer/hubris

[2] https://github.com/oxidecomputer/humility

[3] https://oxide-and-friends.transistor.fm/

It's a mix of embedded work and improving the system's tooling (faster builds, debugger support, etc)

Other folks have mentioned this, but it's important to understand the limitations of Rust with respect to safety. In particular: every stack operation is -- at some level -- an unsafe operation as it operates without a bounds check. This isn't Rust's fault per se; non-segmented architectures don't have an architecturally defined way to know the stack base. As a result, even an entirely safe Rust program can make an illegal access to memory that results in fatal program failure. That, of course, assumes memory protection; if you don't have memory protection (or, like many embedded operating systems, you don't make use of it), stack overflows will plow into adjacent memory.

But wait, it gets worse: stack overflows are often not due to infinite stack consumption (e.g., recursion) but rather simply going deep on an unusual code path. If stack consumption just goes slightly beyond the base of the stack and there is no memory protection, this is corrupt-and-run -- and you are left debugging a problem that looks every bit like a gnarly data race in an unsafe programming language. And this problem becomes especially acute when memory is scarce: you really don't want a tiny embedded system to be dedicating a bunch of its memory to stack space that will never ("never") be used, so you make the stacks as tight as possible -- making stack overflows in fact much more likely.

Indeed, even with the MPU, these problems were acute in the development of Hubris: we originally put the stack at the top of a task's data space, and its data at the bottom -- and we found that tasks that only slightly exceeded their stack (rather than running all of the way through its data and into the protection boundary) were corrupting themselves with difficult-to-debug failures. We flipped the order to assure that every stack overflow hit the protection boundary[0], which required us to be much more intentional about the stack versus data split -- but had the added benefit of allowing us to add debugging support for it.[1]

Stack overflows are still pesky (and still a leading cause of task death!), but without the MPU, each one of these stack overflows would be data corruption -- answering for us viscerally what we "need the MPU for."

[0] https://github.com/oxidecomputer/hubris/commit/d75e832931f67...

[1] https://github.com/oxidecomputer/humility#humility-stackmarg...

In addition to Cliff's talk/blog -- which are absolutely outstanding -- I would recommend listening to the Twitter Space we did on Hubris and Humility last week.[0] It was a really fun conversation, and it also serves as a bit of a B-side for the talk in that it goes into some of the subtler details that we feel are important, but didn't quite rise to the level of the presentation. And of course, be sure to check out the source itself![1][2]

[0] https://www.youtube.com/watch?v=cypmufnPfLw

[1] https://github.com/oxidecomputer/hubris

[2] https://github.com/oxidecomputer/humility

Humility (the debugger)

job "vector" { datacenters = ["dc1"] # system job, runs on all nodes type = "system" group "vector" { count = 1 network { port "api" { to = 8686 } } ephemeral_disk { size = 500 sticky = true } task "vector" { driver = "docker" config { image = "timberio/vector:0.30.0-debian" ports = ["api"] volumes = ["/var/run/docker.sock:/var/run/docker.sock"] } env { VECTOR_CONFIG = "local/vector.toml" VECTOR_REQUIRE_HEALTHY = "false" } resources { cpu = 100 # 100 MHz memory = 100 # 100MB } # template with Vector's configuration template { destination = "local/vector.toml" change_mode = "signal" change_signal = "SIGHUP" # overriding the delimiters to [[ ]] to avoid conflicts with Vector's native templating, which also uses {{ }} left_delimiter = "[[" right_delimiter = "]]" data=<

we are doing something similar with OTEL but we are looking at using https://vector.dev/

We don't pull logs, we forward logs to a centralized logging service.

opensearch - amazon fork of Elasticsearch https://opensearch.org/docs/latestif you do this an have distributed log sources you'd use logstash for, bin off logstash and use vector (https://vector.dev/) its better out of the box for SaaS stuff.

I have done something similar in the past: you can send the logs through a centralized syslog servers (I suggest syslog-ng) and from there ingest into ELK. For parsing I am advice to use something like Vector, is a lot more faster than logstash. When you have your logs ingested correctly, you can create your own dashboard in Kibana. If this fit your requirements, no need to install nginx (unless you want to use as reverse proxy for Kibana), php and mysql.

I think there's nothing currently that combines both logging and metrics into one easy package and visualizes it, but it's also something I would love to have.

Vector[1] would work as the agent, being able to collect both logs and metrics. But the issue would then be storing it. I'm assuming the Elastic Stack might now be able to do both, but it's just to heavy to deal with in a small setup.

A couple of months ago I took a brief look at that when setting up logging for my own homelab (https://pv.wtf/posts/logging-and-the-homelab). Mostly looking at the memory usage to fit it on my synology. Quickwit[2] and Log-Store[3] both come with built in web interfaces that reduce the need for grafana, but neither of them do metrics.

- [1] https://vector.dev

Log to stdout/stderr and collect your logs with a tool like vector (vector.dev) and send it to something like Grafana Loki.

I would recommend that you run vector as a systems service so you don't have to worry about managing it. Here is a basic config to do that - https://github.com/vectordotdev/vector/blob/master/distribution/systemd/vector.service .