ort VS Rocket

To solve this, we built a native extension in Edge Runtime that enables using ONNX runtime via the Rust interface. This was made possible thanks to an excellent Rust wrapper called Ort:

hey hn, supabase ceo here

As the post points out, this comes in 2 parts:

1. Embeddings models for RAG workloads (specifically pgvector). Available today.

2. Large Language Models for GenAI workloads. This will be progressively rolled out as we get our hands on more GPUs.

We've always had a focus on architectures that can run anywhere (especially important for local dev and self-hosting). In that light, we've found that the Ollama[0] tooling is really unbeatable. I heard one of our engineers explain it like "docker for models" which I think is apt.

To support models that work best with GPUs, we're running them with Fly GPUs - pretty much this: https://fly.io/blog/scaling-llm-ollama (and then we stitch a native API around it). The plan is that you will be able to "BYO" model server and point the Edge Runtime towards it using simple env vars / config.

We've also made improvements for CPU models. We built a native extension in Edge Runtime that enables using ONNX runtime via the Rust interface. This was made possible thanks to an excellent Rust wrapper, Ort[1]. We have the models stored on disk, so there is no downloading, cold-boot, etc.

The thing I most like about this set up is that you can now use Edge Functions like background workers for your Postgres database, offloading heavy compute for generating embeddings. For example, you can trigger the worker when a user inserts some text, and then the worker will asynchronously create the embedding and store it back into your database.

I'll be around if there are any questions.

[0] ollama.com

[1] Ort: https://github.com/pykeio/ort

In the quest for more efficient solutions, the ONNX runtime emerged as a beacon of performance. The decision to transition from Typescript to Rust was an unconventional yet pivotal one. Driven by Rust's robust parallel processing capabilities using Rayon and seamless integration with ONNX through the ort crate, Repo-Query unlocked a realm of unparalleled efficiency. The result? A transformation from sluggish processing to, I have to say it, blazing-fast performance.

ort - ONNX runtime library.

Our main model in Rust is a deep neural network, using ONNX via the ort rust bindings. The application is some particular applications of process automation.

You could try ort https://github.com/pykeio/ort It looks like it's in active development and supports GPU inference

I would't try to distribute your ml models with the typical frameworks, especially not with python. Have you looked in to ONNX?For example: https://github.com/pykeio/ort

4. Rocket

If you want something simpler/more minimal, you could use https://rocket.rs/ for the backend and handle the front-end however you want.

I’m the author of https://kellnr.io. When I started working on Kellnr three years ago, https://rocket.rs was “the web framework” to use. Unfortunately, the project seems dead. Before adding more functionality using an unmaintained framework, I want to port Kellnr to https://github.com/tokio-rs/axum.

Or else you could of course just use https://rocket.rs/

Rust is a popular system programming language, known for its robust memory safety features and exceptional performance. While Rust was originally a system programming language, its application has evolved. Now you can see Rust in different app platforms, mobile apps, and of course, in web apps — both in the frontend and backend, with frameworks like Rocket, Axum, and Actix making it even easier to build web applications with Rust.

I've done some research but I have to admit that creating embedded devices is a totally new subject for me, but that is the point of the project - main goal is learning, and creating something is the secondary goal, so please bear with me and my knowledge of the subject. So, for the hardware I've seen many people recommending SMT32 family devices, but I've also read that anything with the Cortex-M processor can be suitable. Need more info on that. OS is a hard choice for me because on one hand I was thinking of Ubuntu Core but the device support is not really that good I think, so other options I've found are Tock and RIOT-OS, and I am gravitating towards the latter because it's main focus is on IOT devices. I've found frameworks like Rocket.rs for a web app, tauri.app for desktop app (which might not be needed but I still like the idea). Also found Tokio.rs which apparently will help with the networking. There was a discussion from the other members about using the Golioth cloud platform with Zephyr and C++, and I don't know if there are any other alternatives for Golioth that support Rust, I've found webthings.io but I am not sure if it's an alternative, or something else actually, so I would be happy to learn more about that. Again I want to hear your recommendations regarding anything that will help creating a project like that.

I use Rocket on the backend with Postgres. Currently experimenting with Yew for the frontend.