ort VS langchainjs

To solve this, we built a native extension in Edge Runtime that enables using ONNX runtime via the Rust interface. This was made possible thanks to an excellent Rust wrapper called Ort:

hey hn, supabase ceo here

As the post points out, this comes in 2 parts:

1. Embeddings models for RAG workloads (specifically pgvector). Available today.

2. Large Language Models for GenAI workloads. This will be progressively rolled out as we get our hands on more GPUs.

We've always had a focus on architectures that can run anywhere (especially important for local dev and self-hosting). In that light, we've found that the Ollama[0] tooling is really unbeatable. I heard one of our engineers explain it like "docker for models" which I think is apt.

To support models that work best with GPUs, we're running them with Fly GPUs - pretty much this: https://fly.io/blog/scaling-llm-ollama (and then we stitch a native API around it). The plan is that you will be able to "BYO" model server and point the Edge Runtime towards it using simple env vars / config.

We've also made improvements for CPU models. We built a native extension in Edge Runtime that enables using ONNX runtime via the Rust interface. This was made possible thanks to an excellent Rust wrapper, Ort[1]. We have the models stored on disk, so there is no downloading, cold-boot, etc.

The thing I most like about this set up is that you can now use Edge Functions like background workers for your Postgres database, offloading heavy compute for generating embeddings. For example, you can trigger the worker when a user inserts some text, and then the worker will asynchronously create the embedding and store it back into your database.

I'll be around if there are any questions.

[0] ollama.com

[1] Ort: https://github.com/pykeio/ort

In the quest for more efficient solutions, the ONNX runtime emerged as a beacon of performance. The decision to transition from Typescript to Rust was an unconventional yet pivotal one. Driven by Rust's robust parallel processing capabilities using Rayon and seamless integration with ONNX through the ort crate, Repo-Query unlocked a realm of unparalleled efficiency. The result? A transformation from sluggish processing to, I have to say it, blazing-fast performance.

ort - ONNX runtime library.

Our main model in Rust is a deep neural network, using ONNX via the ort rust bindings. The application is some particular applications of process automation.

You could try ort https://github.com/pykeio/ort It looks like it's in active development and supports GPU inference

I would't try to distribute your ml models with the typical frameworks, especially not with python. Have you looked in to ONNX?For example: https://github.com/pykeio/ort

*** all code examples are using LangChain TS on the main branch on September 22nd, 2023 (roughly version 0.0.153).

At the time of the prototype's development, the Langchain GitHub loader sent one request per file to fetch the repository sequentially, leading to prolonged download times. In our case about 2 minutes for the insights.opensauced.pizza repository. This issue was later resolved in hwchase17/langchainjs#2224, enabling parallel requests for faster retrieval.

Another llm orchestration library for js/ts

You will probably need to create a custom agent with custom tools to do what you want to do. Look at Langchain (seems like there is an open PR for Google calendar tools here: https://github.com/hwchase17/langchainjs/pull/1777). There are a lot of great integration examples on their website (including for vectorDB memory https://python.langchain.com/docs/modules/memory/how_to/vectorstore_retriever_memory)

You can checkout there docs (here)[https://js.langchain.com/docs/].

The strategy I tried, was to point scalablytyped, at langchainJS.

TL;DR the BANGER TWEET BRAINSToRMER 💥 🐦 🧠 is an open-source, fullstack React/Express/Postgres/Pinecone app that brainstorms new ideas and tweet drafts based on your own notes/ideas and the tweets of your favorite twitter users. This isn't a bot, but you can think of it rather as your personal twitter intern that monitors current twitter **trends**, keeps note of your **ideas**, helps you **brainstorm** new ones, and write **draft** tweets. It's your job to find and edit the best ideas before saving them to your personal notes database or tweeting them out from the app itself. it uses pg-boss cron jobs via Wasp, OpenAI, langChain, and Pinecone for the vector store https://github.com/vincanger/twitter-brainstorming-agent

It is not great. It has a lot of limitations, but can be used under certain conditions. https://github.com/hwchase17/langchainjs/pull/655