q_compress 0.7: still has 35% higher compression ratio than .zstd.parquet for numerical sequences, now with delta encoding and 2x faster than before

• gdal

  44 4,484 10.0 C++

  GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.

  

GDAL is also super useful for converting types for instance converting a netcdf to float32 tiff can be done with

• x3-rust

  1 18 10.0 Rust

  X3 Lossless Audio Compression for Rust

I also had a quick look and compared it against the X3 protocol (similar to FLAC, but more lightweight). q_compress works well in some cases (very low noise and very high noise), while X3 does better in the middle.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• TurboPFor

  8 743 8.5 C

  Fastest Integer Compression

I'm the author of TurboPFor-Integer-Compression. Q_compress is a very interresting project, unfortunatelly it's difficult to compare it to other algorithms. There is not binary or test data files (with q_compress results) available for a simple benchmark. Speed comparison would also be helpfull.

• pcodec

  19 243 8.8 Rust

  Lossless compressor and decompressor for numerical data using quantiles

Here's how you can generate benchmark data, including binary files: https://github.com/mwlon/quantile-compression/blob/main/q_compress/examples/primary.md

• encoding

  1 128 0.0 Go

  Integer Compression Libraries for Go (by zentures)

  

I tried q_compress out on some of the datasets you linked and got these compressed sizes:

• ans-large-alphabet

  1 4 10.0 C++

  Large-Alphabet Semi-Static Entropy Coding Via Asymmetric Numeral Systems

I see the usage field of your algorithm q_compress more in large alphabet integer compression (like lz77 offsets). If you have time, you can generate and test your algorithm with this practical dataset.

Suggest a related project