Any job processing framework like Spark but in Rust?

1. datafusion-ballista

   1 13 1,796 9.1 Rust

   Apache DataFusion Ballista Distributed Query Engine

   

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. fluvio

   2 35 4,960 9.4 Rust

   🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

   

   What are you trying to accomplish? If you are looking for capturing data and transforming on stream and apply time bound calculations, check out: https://github.com/infinyon/flu

4. polars

   3 151 34,403 10.0 Rust

   Dataframes powered by a multithreaded, vectorized query engine, written in Rust

   

   For data frames built on Apache Arrow and: https://github.com/pola-rs/polars/

5. lance

   4 14 5,073 9.8 Rust

   Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

   

   For Feature Stores check out: https://github.com/eto-ai/lance

Suggest a related project