q_compress 0.7: still has 35% higher compression ratio than .zstd.parquet for numerical sequences, now with delta encoding and 2x faster than before

• gdal

  23 3,554 10.0 C++

  GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.

  

  GDAL is also super useful for converting types for instance converting a netcdf to float32 tiff can be done with

• x3-rust

  1 15 10.0 Rust

  X3 Lossless Audio Compression for Rust

  I also had a quick look and compared it against the X3 protocol (similar to FLAC, but more lightweight). q_compress works well in some cases (very low noise and very high noise), while X3 does better in the middle.

• Sonar

  www.sonarsource.com

  sponsored

  Write Clean C++ Code. Always.. Sonar helps you commit clean C++ code every time. With over 550 unique rules to find C++ bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  

• TurboPFor

  3 650 2.0 C

  Fastest Integer Compression

  I'm the author of TurboPFor-Integer-Compression. Q_compress is a very interresting project, unfortunatelly it's difficult to compare it to other algorithms. There is not binary or test data files (with q_compress results) available for a simple benchmark. Speed comparison would also be helpfull.

• quantile-compression

  12 164 6.9 Rust

  Lossless compressor and decompressor for numerical data using quantiles

  Here's how you can generate benchmark data, including binary files: https://github.com/mwlon/quantile-compression/blob/main/q_compress/examples/primary.md

• encoding

  1 123 0.0 Go

  Integer Compression Libraries for Go (by zentures)

  

  I tried q_compress out on some of the datasets you linked and got these compressed sizes:

• ans-large-alphabet

  1 3 10.0 C++

  Large-Alphabet Semi-Static Entropy Coding Via Asymmetric Numeral Systems

  I see the usage field of your algorithm q_compress more in large alphabet integer compression (like lz77 offsets). If you have time, you can generate and test your algorithm with this practical dataset.

Suggest a related project