librdkafka VS hubris

recent PR: https://github.com/confluentinc/librdkafka/pull/4275

# Kafka configuration # https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md bootstrap.servers= security.protocol=SASL_SSL sasl.mechanisms=PLAIN sasl.username= sasl.password= compression.type=gzip compression.level=9 statistics.interval.ms=1000

But what about Kafka producer clients in other languages? The excellent librdkafka project is a C/C++ implementation of Kafka clients and is widely used for non-JVM Kafka applications. Additionally, Kafka clients in other languages (Python, C#) build on top of it. The default partitioner for librdkafka uses the CRC32 hash function to get the correct partition for a key.

We could have made some changes at the librdkafka level (see this), but we didn’t really want to pursue this (at least not yet).

You want records with the same key to always land on the same partition, so you need all the clients to use the same hashing algorithm. The easiest way to do that is to make sure the librdkafka client uses the java compatible murmur2_random hash algorithm. See “Partitioner” section here: https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md

As my first "real world" (ish) project in Vlang, I'm trying to copy https://github.com/confluentinc/confluent-kafka-go, which is a Go wrapper for Kafka C client library, https://github.com/edenhill/librdkafka

If you're using Kafka in a Node.js app, it's likely that you'll need node-rdkafka. This is a library that wraps the librdkafka library and makes it available in Node.js. According to the project's README, "All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library."

# Parse the configuration. # See https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md config_parser = ConfigParser() config_parser.read_file(args.config_file) config = dict(config_parser['default']) # Create Producer instance producer = Producer(config)

• git clone https://github.com/edenhill/librdkafka.git

> The CPUs in Oxide racks are AMD, so, presumably AMD-based compute rather than ARM.

These don’t run Hubris though; based on the chips directory in the repo [0], they’re targeting a mix of NXP and ST parts, which are Arm, and the user isn’t likely to see them or care what firmware they’re running: they’re really pretty “boring”.

[0] : https://github.com/oxidecomputer/hubris/tree/020d014880382d8...

I wouldn't put this comment here. It's not just some detail of this function; it's an invariant of the field that all writers have to respect (maybe this is the only one now but still) and all readers can take advantage of. So I'd add it to the `TaskDesc::regions` docstring. [1]

[1] https://github.com/oxidecomputer/hubris/commit/b44e677fb39cd...

With respect to Hubris, the build badge was, in turns out, pointing to a stale workflow. (That is, the build was succeeding, but the build badge was busted.) This comment has been immortalized in the fix.[0]

With respect to Humility, I am going to resist the temptation of pointing out why one of those directories has a different nomenclature with respect to its delimiter -- and just leave it at this: if you really want to find some filthy code in Humility, you can do much, much better than that!

[0] https://github.com/oxidecomputer/hubris/commit/651a9546b20ce...

A lot of questions in there! Taking these in order:

1. We aren't making standalone servers: the Oxide compute sled comes in the Oxide rack. So are not (and do not intend to be) a drop in replacement for extant rack mounted servers.

2. We have taken a fundamentally different approach to firmware, with a true root of trust that can attest to the service processor -- which can turn attest to the system software. This prompts a lot of questions (e.g., who attests to the root of trust?), and there is a LOT to say about this; look for us to talk a lot more about this

3. In stark contrast (sadly) to nearly everyone else in the server space, the firmware we are developing is entirely open source. More details on that can be found in Cliff Biffle's 2021 OSFC talk and the Hubris and Humility repos.[0][1][2]

4. Definitely not vaporware! We are in the process of shipping to our first customers; you can follow our progress in our Oxide and Friends podcast.[3]

[0] https://www.osfc.io/2021/talks/on-hubris-and-humility-develo...

[1] https://github.com/oxidecomputer/hubris

[2] https://github.com/oxidecomputer/humility

[3] https://oxide-and-friends.transistor.fm/

everywhere, for example in https://github.com/oxidecomputer/hubris/search?q=dyn

Is Box really allocating here? Is the "Rust By Example" text incomplete?

Then I had to stop learning Rust for other reasons, but this doubt really hit me at the time.

I work on an embedded OS in Rust (Hubris) that has a very bespoke build system. As part of the build system, it has to set environmental variables based on (1) the target device and (2) the specific "task"; this is an OS with task-level isolation, so tasks are compiled as individual Rust crates.

Oxide Computer told some storied about the difficulty of bring up of a new motherboard, and mentioned a lot of gotcha details and hack solutions for managing their AMD chip.

They talked about their bring up sequence, boot chain verification on their motherboard, and designing / creating / verifying their hardware root of trust.

I heard mention of this on a podcast recently, trying to find the reference.

I'm pretty sure it was [S3]

- "Tales from the Bringup Lab" https://lnns.co/FBf5oLpyHK3

- or "More Tales from the Bringup Lab" https://lnns.co/LQur_ToJX9m

But I found again these interesting things worth sharing on that search. https://oxide.computer/blog/hubris-and-humility, https://github.com/oxidecomputer/hubris

Search 1 [S1], Trammell Hudson ep mentioning firmware (chromebook related iirc) https://lnns.co/pystdPm0QvG.

Search 2 [S2], Security, Cryptography, Whatever podcast episode mentioning Oxide and roots of trust or similar. https://lnns.co/VnyTvdhBiGC

Search links:

[S1]: https://www.listennotes.com/search/?q=oxide+tpm

[S2]: https://www.listennotes.com/search/?q=oxide%20and%20friends%...

[S3]: https://www.listennotes.com/search/?q=oxide%20and%20friends%...

When we started the company, we knew it would be a three year build -- and indeed, our first product is in the final stages of development (i.e. EMC/safety certification). We have been very transparent about our progress along the way[0][1][2][3][4][5][6][7] -- and our software is essentially all open source, so you can follow along there as well.[8][9][10]

If you are asking "does anyone want a rack-scale computer?" the (short) answer is: yes, they do. The on-prem market has been woefully underserved -- and there are plenty of folks who are sick of Dell/HPE/VMware/Cisco, to say nothing of those who are public cloud borne and wondering if they should perhaps own some of their own compute rather than rent it all.

[0] https://oxide-and-friends.transistor.fm/episodes/holistic-bo...

[1] https://oxide-and-friends.transistor.fm/episodes/the-oxide-s...

[2] https://oxide-and-friends.transistor.fm/episodes/bringup-lab...

[3] https://oxide-and-friends.transistor.fm/episodes/more-tales-...

[4] https://oxide-and-friends.transistor.fm/episodes/another-lpc...

[5] https://oxide-and-friends.transistor.fm/episodes/the-pragmat...

[6] https://oxide-and-friends.transistor.fm/episodes/tales-from-...

[7] https://oxide-and-friends.transistor.fm/episodes/the-sidecar...

[8] https://github.com/oxidecomputer/omicron

[9] https://github.com/oxidecomputer/propolis

[10] https://github.com/oxidecomputer/hubris