LZ4, an Extremely Fast Compression Algorithm

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • Scout APM - Less time debugging, more time building
  • OPS - Build and Run Open Source Unikernels
  • SonarQube - Static code analysis for 29 languages.
  • zstd

    Zstandard - Fast real-time compression algorithm

    LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory.

    Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor.

    In general if send data over network zstd is quite profitable. Even network attached disk AWS EBS or AWS S3 it can be a hugely profitable.

  • 7-Zip-zstd

    7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard

    Here’s a fork of the Windows compression tool 7-Zip which has LZ4 support baked in along with some other useful algorithms – the repo has a good comparison of them: https://github.com/mcmilk/7-Zip-zstd/

  • Scout APM

    Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.

  • squashfs-tools-ng

    A new set of tools and libraries for working with SquashFS images

    A while ago I did some simplistic SquashFS pack/unpack benchmarks[1][2]. I was primarily interested in looking at the behavior of my thread-pool based packer, but as a side effect I got a comparison of compressor speed & ratios over the various available compressors for my Debian test image.

    I must say that LZ4 definitely stands out for both compression and uncompression speed, while still being able to cut the data size in half, making it probably quite suitable for life filesystems and network protocols. Particularly interesting was also comparing Zstd and LZ4[3], the former being substantially slower, but at the same time achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my benchmark at least).

    [1] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...

    [2] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...

    [3] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...

  • lz4_flex

    Pure Rust implementation of raw LZ4 compression/decompression.

    I ported the block format to Rust matching the C implementation in performance and ratio.


  • density

    Superfast compression library

    Some interesting and related projects:

    https://github.com/strigeus/ipzip - TCP/IP Packet Compressor with LZ4 support

    https://github.com/centaurean/density - Extremely fast de/compression

  • RocksDB

    A library that provides an embeddable, persistent key-value store for fast storage.

    W.r.t. standard dictionaries, it's something we're interested in, but the fundamental reality of dictionaries is that their effectiveness is strongly tied to their specificity. Put another way, a universal dictionary is a self-contradiction.

    And yes, totally, I know at least RocksDB supports exactly that behavior [0].

    [0] https://github.com/facebook/rocksdb/blob/12f11373554af219c51...

  • LZ4

    Extremely Fast Compression algorithm

    I'm not a fan of the stacked bar charts, I like the table of data for "Benchmarks" on the github source page: https://github.com/lz4/lz4

    It makes it very clear where LZ4 fits into comparisons with compression speed, decompression speed and compression ratio

  • OPS

    OPS - Build and Run Open Source Unikernels. Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.

  • zfs

    OpenZFS on Linux and FreeBSD

    > zstd is now standard and the default on openzfs 2.0, replacing lz4.

    Are you sure? The default compression level has always been "off", but when switched on - the default has been lz4 for about 5 years. Zstd support was added less than a year ago and there are still a lot of things that need to be fixed before one could even suggest that it might be a sane default. I like zstd, but I like my uncorrupted data more. I know that compatibility between compressor versions and pools is a concern, and there are also the compression performance problems with the way zstd handles zfs block sizes. Thankfully lz4 works great for zfs and has for many years now.


NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts