I built Quantile Compression, which could make all our numerical columnar data 25% smaller.

This page summarizes the projects mentioned and recommended in the original post on /r/dataengineering

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • pcodec

    Lossless compressor and decompressor for numerical data using quantiles

  • You can try it out very easily with the CLI which works on CSV and Parquet columns now, e.g. cargo run --release compress --csv my.csv --col-name my_column out.qco

  • spark-pancake-connector

    support for the "pancake" format in Spark

  • Yep. You can run the docker image and then either use the Spark connector or the Rust client to write to it. I've seen as high as 50k writes/second from one EC2 instance to another. Let me know how it goes!

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Pagefind, a static open-source search library

    1 project | news.ycombinator.com | 9 May 2024
  • Ask HN: How's your experience with Compose/Kotlin multiplatform?

    1 project | news.ycombinator.com | 9 May 2024
  • Aya Rust tutorial Part One

    1 project | dev.to | 9 May 2024
  • Machine learning in Elixir is production-ready

    2 projects | news.ycombinator.com | 9 May 2024
  • Solving the Localhost Development Headache with Nanocl

    2 projects | dev.to | 9 May 2024