ort VS sharp

To solve this, we built a native extension in Edge Runtime that enables using ONNX runtime via the Rust interface. This was made possible thanks to an excellent Rust wrapper called Ort:

hey hn, supabase ceo here

As the post points out, this comes in 2 parts:

1. Embeddings models for RAG workloads (specifically pgvector). Available today.

2. Large Language Models for GenAI workloads. This will be progressively rolled out as we get our hands on more GPUs.

We've always had a focus on architectures that can run anywhere (especially important for local dev and self-hosting). In that light, we've found that the Ollama[0] tooling is really unbeatable. I heard one of our engineers explain it like "docker for models" which I think is apt.

To support models that work best with GPUs, we're running them with Fly GPUs - pretty much this: https://fly.io/blog/scaling-llm-ollama (and then we stitch a native API around it). The plan is that you will be able to "BYO" model server and point the Edge Runtime towards it using simple env vars / config.

We've also made improvements for CPU models. We built a native extension in Edge Runtime that enables using ONNX runtime via the Rust interface. This was made possible thanks to an excellent Rust wrapper, Ort[1]. We have the models stored on disk, so there is no downloading, cold-boot, etc.

The thing I most like about this set up is that you can now use Edge Functions like background workers for your Postgres database, offloading heavy compute for generating embeddings. For example, you can trigger the worker when a user inserts some text, and then the worker will asynchronously create the embedding and store it back into your database.

I'll be around if there are any questions.

[0] ollama.com

[1] Ort: https://github.com/pykeio/ort

In the quest for more efficient solutions, the ONNX runtime emerged as a beacon of performance. The decision to transition from Typescript to Rust was an unconventional yet pivotal one. Driven by Rust's robust parallel processing capabilities using Rayon and seamless integration with ONNX through the ort crate, Repo-Query unlocked a realm of unparalleled efficiency. The result? A transformation from sluggish processing to, I have to say it, blazing-fast performance.

ort - ONNX runtime library.

Our main model in Rust is a deep neural network, using ONNX via the ort rust bindings. The application is some particular applications of process automation.

You could try ort https://github.com/pykeio/ort It looks like it's in active development and supports GPU inference

I would't try to distribute your ml models with the typical frameworks, especially not with python. Have you looked in to ONNX?For example: https://github.com/pykeio/ort

Last thing left is to use our new upload function in our server action. Since I like to upload images in single format and have some more control over them, I will additionally use sharp library. For file name, I'll generate some random string using nanoid:

blur is a technique to blur images while reducing the file size surprisingly. blur works by enlarging the pixels of the image, which reduces the details of the image, and the number of colors also decreases, thus saving storage space. Sharp is a popular image processing library in Node.js, and it supports the blur function. After going through the blur function, the image size at this point is only a few KB, which is reasonable for an image placeholder in the article.

sharp

For simplicity, I will be generating PNGs with JavaScript/Node and the Sharp image library. Any library that can convert between pixel arrays and image files will make the process quite straightforward.

There is also a library that Next.js itself uses: sharp. It can be setup as Node.js service. I even played around a little: image-proxy-service

Sharp Documentation: Link

ESLint does an amazing job in detecting floating promises. I've not had it miss one, ever. When adding this to a project, I've discovered multiple accidental bugs due to a missing "await" keyword--bugs that were extremely subtle and intermittent in many cases.

The only thing it can't do is determine that you actually did handle the promise later. Which is fine. It's a LINTING RULE, and false positives are the name of the game.

What's BAD is when you accidentally miss handling a promise at all. It's an invisible error without the linting rule.

Your other comments...don't even make sense. You're going to build a Lanczos filter by hand? Or you're only going to ... compile ImageMagick to WebAssembly?!, ... an implementation which is tremendously slower (nearly unusably so for large images) than that of Sharp:

https://www.npmjs.com/package/sharp

... which is simply an import away?

No, what you're doing is called "motivated reasoning." You've concluded that Deno is the best, and you're reinterpreting all of my complaints in convoluted ways to support your predetermined conclusion.

Standard fanboy behavior. Or troll behavior. I cite Poe's Law as why it's impossible to tell the difference.