C Compression

Open-source C projects categorized as Compression

Top 23 C Compression Projects

  • zstd

    Zstandard - Fast real-time compression algorithm

    Project mention: Optimizating Rabin-Karp Hashing | news.ycombinator.com | 2024-03-09

    Compression, synchronization and backup systems often use rolling hash to implement "content-defined chunking", an effective form of deduplication.

    In optimized implementations, Rabin-Karp is likely to be the bottleneck. See for instance https://github.com/facebook/zstd/pull/2483 which replaces a Rabin-Karp variant by a >2x faster Gear-Hashing.

  • LZ4

    Extremely Fast Compression algorithm

    Project mention: Number sizes for LZ77 compression | /r/compression | 2023-04-30

    LZ4 is a bit more complicated, but seems faster: https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • ZLib

    A massively spiffy yet delicately unobtrusive compression library.

    Project mention: Zlib 1.3.1 Out | news.ycombinator.com | 2024-01-22
  • cute_headers

    Collection of cross-platform one-file C/C++ libraries with no dependencies, primarily used for games

  • opus

    Modern audio compression for the internet.

    Project mention: Permutation Iteration and Random Access | news.ycombinator.com | 2023-08-23

    There is a pattern here (that also goes with the author's prior article on inverting gauss' sum formula): Generally if if you can make a formula that counts the combination of things you can convert that into a code to encode and decode those combinations into indexes.

    So for example the opus audio codec needs to encode/decode vectors of dimension n whos absolute values sum to k. https://github.com/xiph/opus/blob/master/celt/cwrs.c#L74

    Or this rolling cuckoo filter that optimally encode/decode four sorted numbers in a range 0..2N with the constraint that the they span a range of N. https://github.com/sipa/bitcoin/blob/202006_cuckoo_filter/sr...

    If you're lucky there will be closed form expressions for the encoding and decoding equations. (There for both of the above, at least for some parameters, but in both those examples the implementations use small tables because for the ranges involved the tables end up being faster than sqrts).

  • LZFSE

    LZFSE compression library and command line tool

  • cstore_fdw

    Columnar storage extension for Postgres built as a foreign data wrapper. Check out https://github.com/citusdata/citus for a modernized columnar storage implementation built as a table access method.

    Project mention: Moving a Billion Postgres Rows on a $100 Budget | news.ycombinator.com | 2024-02-21

    Columnar store PostgreSQL extension exists, here are two but I think I’m missing at least another one:

    https://github.com/citusdata/cstore_fdw

    https://github.com/hydradatabase/hydra

    You can also connect other stores using the foreign data wrappers, like parquet files stored on an object store, duckdb, clickhouse… though the joins aren’t optimised as PostgreSQL would do full scan on the external table when joining.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • zlib-ng

    zlib replacement with optimizations for "next generation" systems.

    Project mention: Show HN: Pzip- blazing fast concurrent zip archiver and extractor | news.ycombinator.com | 2023-09-24

    Please note that allowing for 2% bigger resulting file could mean huge speedup in these circumstances even with the same compression routines, seeing these benchmarks of zlib and zlib-ng for different compression levels:

    https://github.com/zlib-ng/zlib-ng/discussions/871

    IMO the fair comparison of the real speed improvement brought by a new program is only between the almost identical resulting compressed sizes.

  • zip

    A portable, simple zip library written in C (by kuba--)

  • FiniteStateEntropy

    New generation entropy codecs : Finite State Entropy and Huff0

    Project mention: Intel QuickAssist Technology Zstandard Plugin for Zstandard | news.ycombinator.com | 2023-08-16

    It's obsolete. It's limited to 32KB LZ window with huffman coding. Zstd can use a much larger window (8MB recommended) and a much better entropy coder: https://github.com/Cyan4973/FiniteStateEntropy

  • Minizip-ng

    Fork of the popular zip manipulation library found in the zlib distribution.

  • smaz

    Small strings compression library

  • c-blosc

    A blocking, shuffling and loss-less compression library that can be faster than `memcpy()`.

  • astc-encoder

    The Arm ASTC Encoder, a compressor for the Adaptive Scalable Texture Compression data format.

  • lzbench

    lzbench is an in-memory benchmark of open-source LZ77/LZSS/LZMA compressors

    Project mention: My experience with btrfs so far | /r/linuxmasterrace | 2023-05-07

    Do not re-compress your file into level 3. The decompression speed is largely the same between level 3 and 8, so you just wasting CPU doing nothing and making your files larger. See the bottom of the README: https://github.com/inikep/lzbench

  • TurboPFor

    Fastest Integer Compression

    Project mention: Show HN: Time Series Benchmark TurboPFor,TurboFloat,TurboFloat LzX,TurboGorilla | news.ycombinator.com | 2023-06-25
  • p7zip

    A new p7zip fork with additional codecs and improvements (forked from https://sourceforge.net/projects/sevenzip/ AND https://sourceforge.net/projects/p7zip/).

    Project mention: Ubuntu 23.04 (Lunar Lobster) | /r/linux | 2023-04-20

    nearly every main distro I am aware of has both available. The reason you still see p7zip is because the CLI incompatibilities vs the newer 7z/7zip executables and the general licensing issues. Most users of "old p7zip" are actually using the actively maintained https://github.com/p7zip-project/p7zip which is updated, supporting unix permissions and zstd and so on.

  • squashfs-tools

    tools to create and extract Squashfs filesystems

  • lizard

    Lizard (formerly LZ5) is an efficient compressor with very fast decompression. It achieves compression ratio that is comparable to zip/zlib and zstd/brotli (at low and medium compression levels) at decompression speed of 1000 MB/s and faster. (by inikep)

  • bzip3

    A better and stronger spiritual successor to BZip2.

  • zson

    ZSON is a PostgreSQL extension for transparent JSONB compression

  • simdcomp

    A simple C library for compressing lists of integers using binary packing

  • precomp-cpp

    Precomp, C++ version - further compress already compressed files

    Project mention: Does epub compress well? | /r/DataHoarder | 2023-05-04

    You should look at Precomp - an apparently abandoned, but awesomely working arhiver that is dedicated to compressing those ZIP-based formats, and compresses them twice-thrice as small as LZMA2 can.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-03-09.

C Compression related posts

Index

What are some of the best open-source Compression projects in C? This list will help you:

Project Stars
1 zstd 21,975
2 LZ4 9,081
3 ZLib 5,190
4 cute_headers 4,061
5 opus 2,056
6 LZFSE 1,748
7 cstore_fdw 1,738
8 zlib-ng 1,425
9 zip 1,306
10 FiniteStateEntropy 1,263
11 Minizip-ng 1,138
12 smaz 1,110
13 c-blosc 953
14 astc-encoder 942
15 lzbench 825
16 TurboPFor 735
17 p7zip 720
18 squashfs-tools 685
19 lizard 633
20 bzip3 629
21 zson 525
22 simdcomp 465
23 precomp-cpp 382
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com