Introducing Kamu - World's first global collaborative data pipeline

• kamu-cli

  7 278 9.7 Rust

  New generation decentralized data lake and a streaming data pipeline

  

We've been developing kamu for 2.5 years, and recently re-wrote it from Scala to Rust - best decision ever :)

• ballista

  20 2,238 9.3 Rust

  Discontinued Distributed compute platform implemented in Rust, and powered by Apache Arrow.

  

In your article you mention looking for a faster data engine, have you looked at Ballista https://github.com/ballista-compute/ballista? It’s pretty young but it uses the Apache Arrow memory model and the maintainer did a bunch of work on Apache Spark I believe.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• open-data-fabric

  3 118 7.5 Python

  Open protocol for decentralized exchange and transformation of data

  

The way kamu already stores metadata makes it very suitable for Blockchain, and thanks to 100% reproducibility - it's not a big stretch to imagine that every time one peer publishes a new data block - a set of other peers could immediately validate the data being published and confirm its validity before even allowing it to commit the new metadata block.

Suggest a related project