q_compress 0.7: still has 35% higher compression ratio than .zstd.parquet for numerical sequences, now with delta encoding and 2x faster than before

This page summarizes the projects mentioned and recommended in the original post on reddit.com/r/rust

Our great sponsors
  • Sonar - Write Clean C++ Code. Always.
  • InfluxDB - Build time-series-based applications quickly and at scale.
  • SaaSHub - Software Alternatives and Reviews
  • gdal

    GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.

    GDAL is also super useful for converting types for instance converting a netcdf to float32 tiff can be done with

  • x3-rust

    X3 Lossless Audio Compression for Rust

    I also had a quick look and compared it against the X3 protocol (similar to FLAC, but more lightweight). q_compress works well in some cases (very low noise and very high noise), while X3 does better in the middle.

  • Sonar

    Write Clean C++ Code. Always.. Sonar helps you commit clean C++ code every time. With over 550 unique rules to find C++ bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  • TurboPFor

    Fastest Integer Compression

    I'm the author of TurboPFor-Integer-Compression. Q_compress is a very interresting project, unfortunatelly it's difficult to compare it to other algorithms. There is not binary or test data files (with q_compress results) available for a simple benchmark. Speed comparison would also be helpfull.

  • quantile-compression

    Lossless compressor and decompressor for numerical data using quantiles

    Here's how you can generate benchmark data, including binary files: https://github.com/mwlon/quantile-compression/blob/main/q_compress/examples/primary.md

  • encoding

    Integer Compression Libraries for Go (by zentures)

    I tried q_compress out on some of the datasets you linked and got these compressed sizes:

  • ans-large-alphabet

    Large-Alphabet Semi-Static Entropy Coding Via Asymmetric Numeral Systems

    I see the usage field of your algorithm q_compress more in large alphabet integer compression (like lz77 offsets). If you have time, you can generate and test your algorithm with this practical dataset.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts