Our great sponsors
Zstandard - Fast real-time compression algorithm
LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory.
Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor.
In general if send data over network zstd is quite profitable. Even network attached disk AWS EBS or AWS S3 it can be a hugely profitable.
7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard
Here’s a fork of the Windows compression tool 7-Zip which has LZ4 support baked in along with some other useful algorithms – the repo has a good comparison of them: https://github.com/mcmilk/7-Zip-zstd/
Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.
A new set of tools and libraries for working with SquashFS images
A while ago I did some simplistic SquashFS pack/unpack benchmarks. I was primarily interested in looking at the behavior of my thread-pool based packer, but as a side effect I got a comparison of compressor speed & ratios over the various available compressors for my Debian test image.
I must say that LZ4 definitely stands out for both compression and uncompression speed, while still being able to cut the data size in half, making it probably quite suitable for life filesystems and network protocols. Particularly interesting was also comparing Zstd and LZ4, the former being substantially slower, but at the same time achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my benchmark at least).
Fastest pure Rust implementation of LZ4 compression/decompression.
I ported the block format to Rust matching the C implementation in performance and ratio.
Superfast compression library
Some interesting and related projects:
https://github.com/strigeus/ipzip - TCP/IP Packet Compressor with LZ4 support
https://github.com/centaurean/density - Extremely fast de/compression
A library that provides an embeddable, persistent key-value store for fast storage.
W.r.t. standard dictionaries, it's something we're interested in, but the fundamental reality of dictionaries is that their effectiveness is strongly tied to their specificity. Put another way, a universal dictionary is a self-contradiction.
And yes, totally, I know at least RocksDB supports exactly that behavior .
Extremely Fast Compression algorithm
I'm not a fan of the stacked bar charts, I like the table of data for "Benchmarks" on the github source page: https://github.com/lz4/lz4
It makes it very clear where LZ4 fits into comparisons with compression speed, decompression speed and compression ratio
Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
OpenZFS on Linux and FreeBSD
> zstd is now standard and the default on openzfs 2.0, replacing lz4.
Are you sure? The default compression level has always been "off", but when switched on - the default has been lz4 for about 5 years. Zstd support was added less than a year ago and there are still a lot of things that need to be fixed before one could even suggest that it might be a sane default. I like zstd, but I like my uncorrupted data more. I know that compatibility between compressor versions and pools is a concern, and there are also the compression performance problems with the way zstd handles zfs block sizes. Thankfully lz4 works great for zfs and has for many years now.
Decompressing the ZST files on Windows tips
2 projects | /r/pushshift | 20 Dec 2022
how do i open/decompress .zst files?
2 projects | /r/PiratedGames | 15 Apr 2022
The LZ4 introduced in PostgreSQL 14 provides faster compression
4 projects | news.ycombinator.com | 8 Nov 2021
How can I fix this
1 project | /r/androidroot | 25 May 2023
Number sizes for LZ77 compression
1 project | /r/compression | 30 Apr 2023