Top 8 Rust Big Data Projects

risingwave

27 6,283 10.0 Rust

Cloud-native SQL stream processing, analytics, and management. KsqlDB and Apache Flink alternative. 🚀 10x more productive. 🚀 10x more cost-efficient.

Project mention: Proton, a fast and lightweight alternative to Apache Flink | news.ycombinator.com | 2024-01-30

How does this compare to RisingWave and Materialize?
https://github.com/risingwavelabs/risingwave

quickwit

64 6,052 9.8 Rust

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.

Project mention: Show HN: Search on S3 Using AWS Lambda | news.ycombinator.com | 2024-01-21

WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
datafusion

55 5,020 9.9 Rust

Apache DataFusion SQL Query Engine

Project mention: Velox: Meta's Unified Execution Engine [pdf] | news.ycombinator.com | 2024-03-25

Python's Substrait seems like the biggest/most-used competitor-ish out there. I'd love some compare & contrast; my sense is that Substrait has a smaller ambition, and more wants to be a language for talking about execution rather than a full on execution engine. https://github.com/substrait-io/substrait
We can also see from the DataFusion discussion that they too see themselves as a bit of a Velox competitor. https://github.com/apache/arrow-datafusion/discussions/6441

Daft

7 1,666 9.8 Rust

Distributed DataFrame for Python designed for the cloud, powered by Rust

Project mention: Daft: Distributed DataFrame for Python | news.ycombinator.com | 2024-02-29

There are benchmarks here - https://github.com/Eventual-Inc/Daft?tab=readme-ov-file#benc.... Seems to outperform Dask by a fair bit.

matano

38 1,354 7.0 Rust

Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS

Project mention: Cisco Acquires Splunk | news.ycombinator.com | 2023-09-21

sorry thats https://matano.dev

datafusion-ballista

12 1,275 8.4 Rust

Apache Arrow Ballista Distributed Query Engine

Project mention: Polars | news.ycombinator.com | 2024-01-08

Not super on topic because this is all immature and not integrated with one another yet, but there is a scaled-out rust data-frames-on-arrow implementation called ballista that could maybe? form the backend of a polars scale out approach: https://github.com/apache/arrow-ballista

blaze

8 883 9.3 Rust

Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core. (by kwai)

Project mention: Blaze: Fast query execution engine for Apache Spark | news.ycombinator.com | 2023-10-19

InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
ReductStore

44 137 9.2 Rust

A time series database for storing and managing large amounts of blob data

Project mention: Time Series Blob Data: ReductStore vs. MongoDB | dev.to | 2024-04-08

In edge computing, managing time series blob data efficiently is critical for performance-sensitive applications. This blog post will compare ReductStore, a specialized time series database for unstructured data, and MongoDB, a widely-used NoSQL database.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).