Lessons Learned from Scaling to Multi-Terabyte Datasets

• iceberg-python

  4 375 9.8 Python

  Apache PyIceberg

  

  Iceberg is working hard to support pure python[0] / rust[1] workflows without Spark. Following Tabular's acquisition [2], I hope it still moves in this direction at the same clip.

  We're using iceberg + duckdb to power analytics in our app[3] and I'm really happy with the combo.

  0 - https://github.com/apache/iceberg-python

  1 - https://github.com/apache/iceberg-rust

  2 - https://x.com/thisritchie/status/1800522255426072647

  3 - https://www.definite.app/

• InfluxDB

  www.influxdata.com featured

  Purpose built for real-time analytics at any scale. InfluxDB Platform is powered by columnar analytics, optimized for cost-efficient storage, and built with open data standards.

  

• iceberg-rust

  2 587 9.4 Rust

  Apache Iceberg

  

  Iceberg is working hard to support pure python[0] / rust[1] workflows without Spark. Following Tabular's acquisition [2], I hope it still moves in this direction at the same clip.

  We're using iceberg + duckdb to power analytics in our app[3] and I'm really happy with the combo.

  0 - https://github.com/apache/iceberg-python

  1 - https://github.com/apache/iceberg-rust

  2 - https://x.com/thisritchie/status/1800522255426072647

  3 - https://www.definite.app/

• delta-rs

  29 2,145 9.8 Rust

  A native Rust library for Delta Lake, with bindings into Python

  

  You can already have it in Delta with Delta Rust and Python bindings: https://github.com/delta-io/delta-rs

• Daft

  10 2,051 9.8 Rust

  Distributed DataFrame for Python designed for the cloud, powered by Rust

  

  https://github.com/Eventual-Inc/Daft Is also great at these types of workloads since it’s both distributed and vectorized!

Suggest a related project