Our great sponsors
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
PeaZip
Free Zip / Unzip software and Rar file extractor. Cross-platform file and archive manager. Features volume spanning, compression, authenticated encryption. Supports 7Z, 7-Zip sfx, ACE, ARJ, Brotli, BZ2, CAB, CHM, CPIO, DEB, GZ, ISO, JAR, LHA/LZH, NSIS, OOo, PAQ/LPAQ, PEA, QUAD, RAR, RPM, split, TAR, Z, ZIP, ZIPX, Zstandard.
They have a nice table on that page: https://github.com/facebook/zstd#benchmarks
Looking at that table, I think LZ4 is a winner. The compression ratio is not too far, compression speed is slightly faster, decompression speed is significantly faster, the code is much simpler so the compiled binary is smaller, and the project is unrelated to Facebook.
I needed to compress a TCP stream [0] for a block-storage project and I've been very impressed with zstd. Combined with io_uring, I'm getting less CPU usage and higher throughput to send and compress data than using socat to send uncompressed data.
> The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.
lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194
There's also LZ4_HC.
There is a reasonably feature complete implementation of Zstd for Go: https://github.com/klauspost/compress/tree/master/zstd
It may not offer the same API 1:1, but it has no interoperability issues that I've encountered. So, I just think no one has bothered to implement it in Rust because most use cases don't mind the added bloat you're talking about. Plus, other comments I've seen suggest that you can actually tune the size of the zstd library, although I'm not sure if the Rust bindings expose that.