Awesome-Rust-MachineLearning VS datafusion

There’s an awesome-git list for a bunch of ML rust stuff not sure how up to date it is as well https://github.com/vaaaaanquish/Awesome-Rust-MachineLearning … not mine

I am looking for something like [Triton Inference Server](https://github.com/triton-inference-server/server) or [TFX Serving](https://www.tensorflow.org/tfx/guide/serving), but in Rust. I came across [Orkon](https://github.com/vertexclique/orkhon) which seems to be dormant and a bunch of examples off of the [Awesome-Rust-MachineLearning](https://github.com/vaaaaanquish/Awesome-Rust-MachineLearning)

There's also https://github.com/vaaaaanquish/Awesome-Rust-MachineLearning

I saw Awesome-Rust-MachineLearning. It have something replacements from Python.

I am working on DAGraph (https://dagraph.com)

DAGraph is a local-first reactive DAG for analytical SQL (OLAP), running entirely in the browser (there is also a native version in the works).

Some tech details: written in Rust, targeting WASM (and native). The SQL engine is Apache DataFusion[1]. The UI uses Egui[2]. Workspace data is persisted in browser using OPFS[3] via OpenDAL[4]. The graph is functional and handles dependencies for you (via parsing the SQL).

Building this to be accessible for beginners while remaining powerful for advanced users. Still very early, lots more features to add, but now usable.

Hope you find it interesting!

[1] https://datafusion.apache.org

Yes! This is a great idea. Thanks for mentioning it! GlueSQL could be another cool target too (https://github.com/gluesql/gluesql). I think there's a fun exploration in taking a storage engine and seeing if it's compatible with different SQL layers — GlueSQL as a simpler starting point, DataFusion as the more complete option (https://github.com/apache/datafusion). Plugging into a real SQL engine seems like a great strategy for uncovering bugs, fixing correctness issues, and discovering what use cases the storage layer actually needs to support. It's also interesting how certain SQL engines and storage engines tend to align well with each other — the right pairing can unlock a lot. Definitely a direction I want to explore.

Large numbers of Equality Delete files can cause significant read amplification, as query engines must scan both data files and delete files and perform value matching. RisingWave internally implements a schedulable compaction service (see https://github.com/nimtable/iceberg-compaction, an engine based on DataFusion project) that periodically removes equality delete files and compacts small files, reducing read amplification and storage fragmentation while preserving data freshness:

I we are actively working on supporting extension types. The mechanism is likely to be using the Arrow extension type mechanism (a logical annotation on top of existing Arrow types https://arrow.apache.org/docs/format/Columnar.html#format-me...)

I expect this to be used to support Variant https://github.com/apache/datafusion/issues/16116

As well as geometry types

While our Go-based implementation has served us well, we've been exploring whether Rust—with its rapidly maturing Ethereum ecosystem—could take us even further. The potential benefits are compelling: better performance, enhanced safety, and improved portability that could make it easier to bring these UDFs to other analytical engines like DataFusion or DuckDB.

Based on Apache DataFusion.

Remote: Yes

Willing to relocate: Yes

Technologies: Rust, Nodejs, Javascript, Typescript, Golang

Résumé/CV: https://drive.google.com/drive/folders/1ecTn700lcmt8cqlnBTtm...

Email: jatin6972@gmail.com

Github: https://github.com/jatin510

Info: Hi, I'm Jagdish Parihar! A Backend Engineer with 4+ years of experience building scalable systems and microservices using Rust, Node.js, and Golang. I've contributed to open-source projects like Apache DataFusion and thrive on solving complex backend challenges.

I'm exploring the opportunity to be working in the DB based startups. I am looking for an entry to be an engineer who will work on databases. Currently, I am contributing to open source, looking for part-time/full-time working with databases.

Datafusion contributions: https://github.com/apache/datafusion/pulls?q=is%3Apr+author%...

Datafusion comet contributions: https://github.com/apache/datafusion-comet/pulls?q=is%3Apr+a...

Let’s connect!