Zstandard – Real-time data compression algorithm

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • Rustup

    The Rust toolchain installer

  • I wish Rust would add this, but they don't seem interested at all:

    https://github.com/rust-lang/rustup/issues/1858

  • caddy-brotli

    Brotli compression encoder for Caddy

  • Just did some quick research and it looks like you can serve pre-compressed static resources in Brotil. See the precompressed option of the file_server directive[1].

    The thinking seems to be due to how CPU intensive it is, Brotil is not favoured for on-the-fly compression[2].

    If you're really desperate for it though, you could try [this extension][3].

    [1] https://caddyserver.com/docs/caddyfile/directives/file_serve...

    [2] https://caddy.community/t/caddy-v2-brotli/8805

    [3] https://github.com/ueffel/caddy-brotli

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • compress

    Optimized Go Compression Packages

  • Recent versions of zstd definitely don't obsolete LZ4, or else I don't think the author would still be contributing to both...

    And if you're going to play with Snappy, you might find S2, which was linked on HN relatively recently, interesting. [1]

    [1] - https://github.com/klauspost/compress/tree/master/s2

  • zstd

    Zstandard - Fast real-time compression algorithm

  • Yes, sort of: "Zstd offers a training mode, which can be used to tune the algorithm for a selected type of data. Training Zstandard is achieved by providing it with a few samples (one file per sample). The result of this training is stored in a file called "dictionary", which must be loaded before compression and decompression. Using this dictionary, the compression ratio achievable on small data improves dramatically."

    https://github.com/facebook/zstd#the-case-for-small-data-com...

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts