Top 20 C Compression Projects

• zstd

  4 13,801 9.7 C

  Zstandard - Fast real-time compression algorithm

  Latest mention: Linus Torvalds on the importance of ECC in main memory (RAM) | news.ycombinator.com | 2021-01-03

  Yes. I work at Facebook on data compression.

  The libraries we maintain (1) are responsible for a non-trivial part of Facebook's overall compute footprint, (2) should basically never fail of their own accord, and (3) have pretty good error monitoring. So my team is operating what is effectively (among other things) a very sensitive detector for hardware failure.

  And indeed we see examples all the time of blobs that fail to decompress, and usually when we dig in we find that the blob is only a single bit-flip away from a blob that decompresses successfully into a syntactically correct message. I can't share numbers, but, off the top of my head, I think it's the largest source of failures we see. It happens frequently enough that I wrote a tool to automate checking [0].

  So yes. It happens. Pretty frequently, in the sense that if you're doing xillions of operations a day, a one-in-a-xillion failure happens all the time.

  [0] https://github.com/facebook/zstd/tree/dev/contrib/diagnose_c...

• brotli

  1 9,610 6.5 C

  Brotli compression format

  Latest mention: Serving compressed static assets with HTTP in Go 1.16 | dev.to | 2021-01-17

  First, it checks if request accepts compressed encoding of response. If yes, it checks if script.js.gz (or script.js.br for brotli) is available in embedded file system and serves contents of script.js.gz.

• borg

  2 6,891 8.9 C

  Deduplicating archiver with compression and authenticated encryption.

  Latest mention: Encrypted Backup Shootout | news.ycombinator.com | 2021-01-02

  borg (c) - https://github.com/borgbackup/borg

• lz4

  2 5,647 0.0 C

  Extremely Fast Compression algorithm

  Latest mention: lz4_flex 0.7.2 reaches parity with cpp reference implementation on speed and ratio | reddit.com/r/rust | 2021-01-18

  It's hard to tell, how much since the latest lz4 1.9 has been improved, because the (benchmarks)[https://github.com/lz4/lz4/commit/31763c59fca9a38bc697cd37f591d6ba385f7614#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5] are executed with different processors

• zlib

  0 2,738 0.0 C

  A massively spiffy yet delicately unobtrusive compression library.

• lzfse

  1 1,578 0.0 C

  LZFSE compression library and command line tool

  Latest mention: Why is Zip preferred on Mac while RAR seems like the favorite on Windows? | reddit.com/r/MacOS | 2021-01-21

  I'm not entirely correct, though. Because Unarchiver likely uses this compression algorithm by calling directly into the API. And there is an open-source implementation for it: lzfse/lzfse.

• opus

  1 1,127 6.4 C

  Modern audio compression for the internet.

  Latest mention: Using Opus in Unity3d. | reddit.com/r/dissonance_voip | 2020-12-31

  If you don't own Dissonance it'll be a little more difficult. You'll need to compile opus into a dll (source code here) and then write a C# wrapper to call the opus methods from C# (using DllImport).

• FiniteStateEntropy

  0 934 3.3 C

  New generation entropy codecs : Finite State Entropy and Huff0

• smaz

  0 875 0.0 C

  Small strings compression library

• minizip

  0 735 8.4 C

  Fork of the popular zip manipulation library found in the zlib distribution.

• zlib-ng

  0 712 9.1 C

  zlib replacement with optimizations for "next generation" systems.

• flate2-rs

  0 378 5.8 C

  DEFLATE, gzip, and zlib bindings for Rust

• python-blosc

  0 261 6.8 C

  A Python wrapper for the extremely fast Blosc compression library

• precomp-cpp

  1 239 3.4 C

  Precomp, C++ version - further compress already compressed files

  Latest mention: Help me out with some compression tips | reddit.com/r/compression | 2021-01-06

• TinyJpeg

  0 138 0.0 C

  Single header lib for JPEG encoding. Public domain. C99. stb style.

• snebu

  1 88 7.8 C

  Simple Network Encrypting Backup Utility

  Latest mention: Encrypted Backup Shootout | news.ycombinator.com | 2021-01-02

  snebu (c) - https://github.com/derekp7/snebu

• apultra

  1 52 8.0 C

  Free open-source compressor for apLib with 5-7% better ratios

  Latest mention: Free open source aPLib compressor gains serious use in retrocomputing | news.ycombinator.com | 2021-01-12

• bzip2-rs

  0 50 3.8 C

  libbz2 (bzip2 compression) bindings for Rust

• MV-Tractus

  1 46 4.1 C

  A simple tool to extract motion vectors from h264 encoded videos.

  Latest mention: Motion detection from video compression | reddit.com/r/computervision | 2021-01-05

  look for libraries to extract motion vector from video streams. e.g. https://github.com/vadimkantorov/mpegflow or https://github.com/jishnujayakumar/MV-Tractus

• lzmat

  0 2 0.0 C

  git mirror of LZMAT (http://www.matcode.com/lzmat.htm)

Index