q_compress 0.7: still has 35% higher compression ratio than .zstd.parquet for numerical sequences, now with delta encoding and 2x faster than before

1. gdal

   1 47 6,013 10.0 C++

   GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.

   

   GDAL is also super useful for converting types for instance converting a netcdf to float32 tiff can be done with

2. AppSignal

   www.appsignal.com featured

   Monitoring that respects your time & budget. APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.

   

3. x3-rust

   2 1 20 7.2 Rust

   X3 Lossless Audio Compression for Rust

   

   I also had a quick look and compared it against the X3 protocol (similar to FLAC, but more lightweight). q_compress works well in some cases (very low noise and very high noise), while X3 does better in the middle.

4. TurboPFor

   3 8 836 9.3 C

   Fastest Integer Compression

   

   I'm the author of TurboPFor-Integer-Compression. Q_compress is a very interresting project, unfortunatelly it's difficult to compare it to other algorithms. There is not binary or test data files (with q_compress results) available for a simple benchmark. Speed comparison would also be helpfull.

5. pcodec

   4 21 494 7.9 Rust

   Lossless codec for numerical data

   

   Here's how you can generate benchmark data, including binary files: https://github.com/mwlon/quantile-compression/blob/main/q_compress/examples/primary.md

6. encoding

   5 1 133 0.0 Go

   Integer Compression Libraries for Go (by zentures)

   

   I tried q_compress out on some of the datasets you linked and got these compressed sizes:

7. ans-large-alphabet

   6 1 4 10.0 C++

   Large-Alphabet Semi-Static Entropy Coding Via Asymmetric Numeral Systems

   

   I see the usage field of your algorithm q_compress more in large alphabet integer compression (like lz77 offsets). If you have time, you can generate and test your algorithm with this practical dataset.

Suggest a related project