LZ4, an Extremely Fast Compression Algorithm

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • SonarQube - Static code analysis for 29 languages.
  • InfluxDB - Access the most powerful time series database as a service
  • SaaSHub - Software Alternatives and Reviews
  • zstd

    Zstandard - Fast real-time compression algorithm

    LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory.

    Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor.

    In general if send data over network zstd is quite profitable. Even network attached disk AWS EBS or AWS S3 it can be a hugely profitable.

  • 7-Zip-zstd

    7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard

    Here’s a fork of the Windows compression tool 7-Zip which has LZ4 support baked in along with some other useful algorithms – the repo has a good comparison of them: https://github.com/mcmilk/7-Zip-zstd/

  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  • squashfs-tools-ng

    A new set of tools and libraries for working with SquashFS images

    A while ago I did some simplistic SquashFS pack/unpack benchmarks[1][2]. I was primarily interested in looking at the behavior of my thread-pool based packer, but as a side effect I got a comparison of compressor speed & ratios over the various available compressors for my Debian test image.

    I must say that LZ4 definitely stands out for both compression and uncompression speed, while still being able to cut the data size in half, making it probably quite suitable for life filesystems and network protocols. Particularly interesting was also comparing Zstd and LZ4[3], the former being substantially slower, but at the same time achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my benchmark at least).

    [1] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...

    [2] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...

    [3] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...

  • lz4_flex

    Fastest pure Rust implementation of LZ4 compression/decompression.

    I ported the block format to Rust matching the C implementation in performance and ratio.


  • density

    Superfast compression library

    Some interesting and related projects:

    https://github.com/strigeus/ipzip - TCP/IP Packet Compressor with LZ4 support

    https://github.com/centaurean/density - Extremely fast de/compression

  • RocksDB

    A library that provides an embeddable, persistent key-value store for fast storage.

    W.r.t. standard dictionaries, it's something we're interested in, but the fundamental reality of dictionaries is that their effectiveness is strongly tied to their specificity. Put another way, a universal dictionary is a self-contradiction.

    And yes, totally, I know at least RocksDB supports exactly that behavior [0].

    [0] https://github.com/facebook/rocksdb/blob/12f11373554af219c51...

  • LZ4

    Extremely Fast Compression algorithm

    I'm not a fan of the stacked bar charts, I like the table of data for "Benchmarks" on the github source page: https://github.com/lz4/lz4

    It makes it very clear where LZ4 fits into comparisons with compression speed, decompression speed and compression ratio

  • InfluxDB

    Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  • zfs

    OpenZFS on Linux and FreeBSD

    > zstd is now standard and the default on openzfs 2.0, replacing lz4.

    Are you sure? The default compression level has always been "off", but when switched on - the default has been lz4 for about 5 years. Zstd support was added less than a year ago and there are still a lot of things that need to be fixed before one could even suggest that it might be a sane default. I like zstd, but I like my uncorrupted data more. I know that compatibility between compressor versions and pools is a concern, and there are also the compression performance problems with the way zstd handles zfs block sizes. Thankfully lz4 works great for zfs and has for many years now.


NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts