LZ4, an Extremely Fast Compression Algorithm

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
  • zstd

    Zstandard - Fast real-time compression algorithm

    LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory.

    Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor.

    In general if send data over network zstd is quite profitable. Even network attached disk AWS EBS or AWS S3 it can be a hugely profitable.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • 7-Zip-zstd

    7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard

    Here’s a fork of the Windows compression tool 7-Zip which has LZ4 support baked in along with some other useful algorithms – the repo has a good comparison of them: https://github.com/mcmilk/7-Zip-zstd/

  • squashfs-tools-ng

    A new set of tools and libraries for working with SquashFS images

    A while ago I did some simplistic SquashFS pack/unpack benchmarks[1][2]. I was primarily interested in looking at the behavior of my thread-pool based packer, but as a side effect I got a comparison of compressor speed & ratios over the various available compressors for my Debian test image.

    I must say that LZ4 definitely stands out for both compression and uncompression speed, while still being able to cut the data size in half, making it probably quite suitable for life filesystems and network protocols. Particularly interesting was also comparing Zstd and LZ4[3], the former being substantially slower, but at the same time achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my benchmark at least).

    [1] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...

    [2] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...

    [3] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...

  • lz4_flex

    Fastest pure Rust implementation of LZ4 compression/decompression.

    I ported the block format to Rust matching the C implementation in performance and ratio.


  • density

    Discontinued Superfast compression library

    Some interesting and related projects:

    https://github.com/strigeus/ipzip - TCP/IP Packet Compressor with LZ4 support

    https://github.com/centaurean/density - Extremely fast de/compression

  • RocksDB

    A library that provides an embeddable, persistent key-value store for fast storage.

    W.r.t. standard dictionaries, it's something we're interested in, but the fundamental reality of dictionaries is that their effectiveness is strongly tied to their specificity. Put another way, a universal dictionary is a self-contradiction.

    And yes, totally, I know at least RocksDB supports exactly that behavior [0].

    [0] https://github.com/facebook/rocksdb/blob/12f11373554af219c51...

  • LZ4

    Extremely Fast Compression algorithm

    I'm not a fan of the stacked bar charts, I like the table of data for "Benchmarks" on the github source page: https://github.com/lz4/lz4

    It makes it very clear where LZ4 fits into comparisons with compression speed, decompression speed and compression ratio

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
  • zfs

    OpenZFS on Linux and FreeBSD

    > zstd is now standard and the default on openzfs 2.0, replacing lz4.

    Are you sure? The default compression level has always been "off", but when switched on - the default has been lz4 for about 5 years. Zstd support was added less than a year ago and there are still a lot of things that need to be fixed before one could even suggest that it might be a sane default. I like zstd, but I like my uncorrupted data more. I know that compatibility between compressor versions and pools is a concern, and there are also the compression performance problems with the way zstd handles zfs block sizes. Thankfully lz4 works great for zfs and has for many years now.


NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Decompressing the ZST files on Windows tips

    2 projects | /r/pushshift | 20 Dec 2022
  • how do i open/decompress .zst files?

    2 projects | /r/PiratedGames | 15 Apr 2022
  • The LZ4 introduced in PostgreSQL 14 provides faster compression

    4 projects | news.ycombinator.com | 8 Nov 2021
  • Zip: How not to design a file format

    2 projects | news.ycombinator.com | 16 Oct 2023
  • How can I fix this

    1 project | /r/androidroot | 25 May 2023

Did you konow that C is
the 7th most popular programming language
based on number of metions?