Test On 4 Concurrent Jobs Using Python-Polars 0.17.11 to GroupBy Billion Rows

• peaks-framework

  8 46 10.0 Go

  Discontinued The Peaks Consolidation is equipped with state-of-the-art algorithms and data structures that support high-performance databending exercises. It specializes in management accounting and consolidation, with some special topics in machine learning and bioinformatics. [Moved to: https://github.com/hkpeaks/peaks-consolidation]

This project has only 3-month history, first trial vesion to be released in Jun, provide most fundamental commands. For further info, you can visit github.com/hkpeaks/peaks-framework

• polars

  144 26,378 10.0 Rust

  Dataframes powered by a multithreaded, vectorized query engine, written in Rust

  

I successfully ran four jobs with a billion rows yesterday while testing trillions of rows for more than a million files using Polars and Peaks on a step-by-step progressive basis. Previously, Polars failed on a single job, but after several bug fixes, it can now handle the workload. You can see https://github.com/pola-rs/polars/issues/7774

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• cloudquery

  102 5,591 10.0 Go

  The open source high performance ELT framework powered by Apache Arrow

  

CloudQuery supports a lot of APIs https://github.com/cloudquery/cloudquery

Suggest a related project