q_compress 0.7: still has 35% higher compression ratio than .zstd.parquet for numerical sequences, now with delta encoding and 2x faster than before

This page summarizes the projects mentioned and recommended in the original post on /r/rust

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • gdal

    GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.

    GDAL is also super useful for converting types for instance converting a netcdf to float32 tiff can be done with

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • x3-rust

    X3 Lossless Audio Compression for Rust

    I also had a quick look and compared it against the X3 protocol (similar to FLAC, but more lightweight). q_compress works well in some cases (very low noise and very high noise), while X3 does better in the middle.

  • TurboPFor

    Fastest Integer Compression

    I'm the author of TurboPFor-Integer-Compression. Q_compress is a very interresting project, unfortunatelly it's difficult to compare it to other algorithms. There is not binary or test data files (with q_compress results) available for a simple benchmark. Speed comparison would also be helpfull.

  • pcodec

    Lossless codec for numerical data

    Here's how you can generate benchmark data, including binary files: https://github.com/mwlon/quantile-compression/blob/main/q_compress/examples/primary.md

  • encoding

    Integer Compression Libraries for Go (by zentures)

    I tried q_compress out on some of the datasets you linked and got these compressed sizes:

  • ans-large-alphabet

    Large-Alphabet Semi-Static Entropy Coding Via Asymmetric Numeral Systems

    I see the usage field of your algorithm q_compress more in large alphabet integer compression (like lz77 offsets). If you have time, you can generate and test your algorithm with this practical dataset.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Building a Dynamic Tile Server Using Cloud Optimized GeoTIFF(COG) with TiTiler

    3 projects | dev.to | 21 Dec 2023
  • Show HN: Time Series Benchmark TurboPFor,TurboFloat,TurboFloat LzX,TurboGorilla

    1 project | news.ycombinator.com | 25 Jun 2023
  • Company decided to move away from AutoCAD to something cheaper...

    1 project | /r/civilengineering | 8 Jun 2023
  • gdal 0.15 is out!

    2 projects | /r/rust | 2 Jun 2023
  • Not sure if I'm ready to make the jump from Unity yet.

    1 project | /r/godot | 10 May 2023

Did you konow that C++ is
the 6th most popular programming language
based on number of metions?