Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR. Learn more →
Top 12 Rust Inference Projects
-
-
CodeRabbit
CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
-
That’s right, if LLMs were really thinking/forming world models etc. we would expect them to be robust against word choice or phrasing. But in practice anyone using RAG can tell you that that is not the case.
I’m just a practitioner so my language might be imprecise but when I say similarly structured sentences what I mean is, and this is my interpretation based on my experience with using Agents and LLMs, that the shape of the context as in the phrasing and the word choice highly bias the outputs of LLMs.
In my own observations at work, those who interpret LLMs to be thinking often produce bad agents. LLM are not good at open ended questions, if you ask an LLM “improve this code” you will often get bad results that just look passable. But if you interpret LLMs as probabilistic models highly biased by their context then you would add a lot more context and specific instructions in the prompt in order to get the Agent to produce the right output.
Side note, this is also why I like the AICI approach: https://github.com/microsoft/aici
-
To solve this, we built a native extension in Edge Runtime that enables using ONNX runtime via the Rust interface. This was made possible thanks to an excellent Rust wrapper called Ort:
-
After spending some months working on the Pipeless open-source framework, today I bring something new and really cool: Pipeless Agents
-
-
-
-
Nutrient
Nutrient - The #1 PDF SDK Library. Bad PDFs = bad UX. Slow load times, broken annotations, clunky UX frustrates users. Nutrient’s PDF SDKs gives seamless document experiences, fast rendering, annotations, real-time collaboration, 100+ features. Used by 10K+ devs, serving ~half a billion users worldwide. Explore the SDK for free.
-
-
-
-
Project mention: Floating Point Precision: Understanding FP64, FP32, and FP16 in Large Language Models | dev.to | 2025-02-08
Recently, I started working on my own inference API as a side project and found myself grappling with torch_dtype , which determines the data type of a Tensor. Each dtype is a floating point and values are usually defined in a model's config.json, but I wanted to dig deeper into how they work. I figured it was worth learning more and sharing my findings with others.
-
Rust Inference discussion
Rust Inference related posts
-
What Is ChatGPT Doing and Why Does It Work?
-
Google Gemini: Context Caching
-
AI Inference Now Available in Supabase Edge Functions
-
Meta's Segment Anything written with C++ / GGML
-
Machine Learning Inference Server in Rust?
-
A note from our sponsor - CodeRabbit
coderabbit.ai | 16 Feb 2025
Index
What are some of the best open-source Inference projects in Rust? This list will help you:
# | Project | Stars |
---|---|---|
1 | typedb | 3,921 |
2 | aici | 1,998 |
3 | ort | 1,119 |
4 | pipeless | 736 |
5 | blindai | 505 |
6 | snips-nlu-rs | 341 |
7 | onnxruntime-rs | 287 |
8 | llama-dfdx | 102 |
9 | Kyanite | 59 |
10 | meteorite | 8 |
11 | fastllm | 5 |
12 | mediapipe-rs | 0 |