ort VS hubris

To solve this, we built a native extension in Edge Runtime that enables using ONNX runtime via the Rust interface. This was made possible thanks to an excellent Rust wrapper called Ort:

hey hn, supabase ceo here

As the post points out, this comes in 2 parts:

1. Embeddings models for RAG workloads (specifically pgvector). Available today.

2. Large Language Models for GenAI workloads. This will be progressively rolled out as we get our hands on more GPUs.

We've always had a focus on architectures that can run anywhere (especially important for local dev and self-hosting). In that light, we've found that the Ollama[0] tooling is really unbeatable. I heard one of our engineers explain it like "docker for models" which I think is apt.

To support models that work best with GPUs, we're running them with Fly GPUs - pretty much this: https://fly.io/blog/scaling-llm-ollama (and then we stitch a native API around it). The plan is that you will be able to "BYO" model server and point the Edge Runtime towards it using simple env vars / config.

We've also made improvements for CPU models. We built a native extension in Edge Runtime that enables using ONNX runtime via the Rust interface. This was made possible thanks to an excellent Rust wrapper, Ort[1]. We have the models stored on disk, so there is no downloading, cold-boot, etc.

The thing I most like about this set up is that you can now use Edge Functions like background workers for your Postgres database, offloading heavy compute for generating embeddings. For example, you can trigger the worker when a user inserts some text, and then the worker will asynchronously create the embedding and store it back into your database.

I'll be around if there are any questions.

[0] ollama.com

[1] Ort: https://github.com/pykeio/ort

In the quest for more efficient solutions, the ONNX runtime emerged as a beacon of performance. The decision to transition from Typescript to Rust was an unconventional yet pivotal one. Driven by Rust's robust parallel processing capabilities using Rayon and seamless integration with ONNX through the ort crate, Repo-Query unlocked a realm of unparalleled efficiency. The result? A transformation from sluggish processing to, I have to say it, blazing-fast performance.

ort - ONNX runtime library.

Our main model in Rust is a deep neural network, using ONNX via the ort rust bindings. The application is some particular applications of process automation.

You could try ort https://github.com/pykeio/ort It looks like it's in active development and supports GPU inference

I would't try to distribute your ml models with the typical frameworks, especially not with python. Have you looked in to ONNX?For example: https://github.com/pykeio/ort

> The CPUs in Oxide racks are AMD, so, presumably AMD-based compute rather than ARM.

These don’t run Hubris though; based on the chips directory in the repo [0], they’re targeting a mix of NXP and ST parts, which are Arm, and the user isn’t likely to see them or care what firmware they’re running: they’re really pretty “boring”.

[0] : https://github.com/oxidecomputer/hubris/tree/020d014880382d8...

I wouldn't put this comment here. It's not just some detail of this function; it's an invariant of the field that all writers have to respect (maybe this is the only one now but still) and all readers can take advantage of. So I'd add it to the `TaskDesc::regions` docstring. [1]

[1] https://github.com/oxidecomputer/hubris/commit/b44e677fb39cd...

With respect to Hubris, the build badge was, in turns out, pointing to a stale workflow. (That is, the build was succeeding, but the build badge was busted.) This comment has been immortalized in the fix.[0]

With respect to Humility, I am going to resist the temptation of pointing out why one of those directories has a different nomenclature with respect to its delimiter -- and just leave it at this: if you really want to find some filthy code in Humility, you can do much, much better than that!

[0] https://github.com/oxidecomputer/hubris/commit/651a9546b20ce...

A lot of questions in there! Taking these in order:

1. We aren't making standalone servers: the Oxide compute sled comes in the Oxide rack. So are not (and do not intend to be) a drop in replacement for extant rack mounted servers.

2. We have taken a fundamentally different approach to firmware, with a true root of trust that can attest to the service processor -- which can turn attest to the system software. This prompts a lot of questions (e.g., who attests to the root of trust?), and there is a LOT to say about this; look for us to talk a lot more about this

3. In stark contrast (sadly) to nearly everyone else in the server space, the firmware we are developing is entirely open source. More details on that can be found in Cliff Biffle's 2021 OSFC talk and the Hubris and Humility repos.[0][1][2]

4. Definitely not vaporware! We are in the process of shipping to our first customers; you can follow our progress in our Oxide and Friends podcast.[3]

[0] https://www.osfc.io/2021/talks/on-hubris-and-humility-develo...

[1] https://github.com/oxidecomputer/hubris

[2] https://github.com/oxidecomputer/humility

[3] https://oxide-and-friends.transistor.fm/

everywhere, for example in https://github.com/oxidecomputer/hubris/search?q=dyn

Is Box really allocating here? Is the "Rust By Example" text incomplete?

Then I had to stop learning Rust for other reasons, but this doubt really hit me at the time.

I work on an embedded OS in Rust (Hubris) that has a very bespoke build system. As part of the build system, it has to set environmental variables based on (1) the target device and (2) the specific "task"; this is an OS with task-level isolation, so tasks are compiled as individual Rust crates.

Oxide Computer told some storied about the difficulty of bring up of a new motherboard, and mentioned a lot of gotcha details and hack solutions for managing their AMD chip.

They talked about their bring up sequence, boot chain verification on their motherboard, and designing / creating / verifying their hardware root of trust.

I heard mention of this on a podcast recently, trying to find the reference.

I'm pretty sure it was [S3]

- "Tales from the Bringup Lab" https://lnns.co/FBf5oLpyHK3

- or "More Tales from the Bringup Lab" https://lnns.co/LQur_ToJX9m

But I found again these interesting things worth sharing on that search. https://oxide.computer/blog/hubris-and-humility, https://github.com/oxidecomputer/hubris

Search 1 [S1], Trammell Hudson ep mentioning firmware (chromebook related iirc) https://lnns.co/pystdPm0QvG.

Search 2 [S2], Security, Cryptography, Whatever podcast episode mentioning Oxide and roots of trust or similar. https://lnns.co/VnyTvdhBiGC

Search links:

[S1]: https://www.listennotes.com/search/?q=oxide+tpm

[S2]: https://www.listennotes.com/search/?q=oxide%20and%20friends%...

[S3]: https://www.listennotes.com/search/?q=oxide%20and%20friends%...

When we started the company, we knew it would be a three year build -- and indeed, our first product is in the final stages of development (i.e. EMC/safety certification). We have been very transparent about our progress along the way[0][1][2][3][4][5][6][7] -- and our software is essentially all open source, so you can follow along there as well.[8][9][10]

If you are asking "does anyone want a rack-scale computer?" the (short) answer is: yes, they do. The on-prem market has been woefully underserved -- and there are plenty of folks who are sick of Dell/HPE/VMware/Cisco, to say nothing of those who are public cloud borne and wondering if they should perhaps own some of their own compute rather than rent it all.

[0] https://oxide-and-friends.transistor.fm/episodes/holistic-bo...

[1] https://oxide-and-friends.transistor.fm/episodes/the-oxide-s...

[2] https://oxide-and-friends.transistor.fm/episodes/bringup-lab...

[3] https://oxide-and-friends.transistor.fm/episodes/more-tales-...

[4] https://oxide-and-friends.transistor.fm/episodes/another-lpc...

[5] https://oxide-and-friends.transistor.fm/episodes/the-pragmat...

[6] https://oxide-and-friends.transistor.fm/episodes/tales-from-...

[7] https://oxide-and-friends.transistor.fm/episodes/the-sidecar...

[8] https://github.com/oxidecomputer/omicron

[9] https://github.com/oxidecomputer/propolis

[10] https://github.com/oxidecomputer/hubris