Compression

Top 23 Compression Open-Source Projects

  • DeepSpeed

    DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

  • Project mention: Can we discuss MLOps, Deployment, Optimizations, and Speed? | /r/LocalLLaMA | 2023-12-06

    DeepSpeed can handle parallelism concerns, and even offload data/model to RAM, or even NVMe (!?) . I'm surprised I don't see this project used more.

  • CyberChef

    The Cyber Swiss Army Knife - a web app for encryption, encoding, compression and data analysis

  • Project mention: PicoCTF 2024: packer | dev.to | 2024-04-05

    Then we take the encrypted text and use CyberChef to decrypt it.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • zstd

    Zstandard - Fast real-time compression algorithm

  • Project mention: Chrome Feature: ZSTD Content-Encoding | news.ycombinator.com | 2024-04-01

    Of course, you may get different results with another dataset.

    gzip (zlib -6) [ratio=32%] [compr=35Mo/s] [dec=407Mo/s]

    zstd (zstd -2) [ratio=32%] [compr=356Mo/s] [dec=1067Mo/s]

    NB1: The default for zstd is -3, but the table only had -2. The difference is probably small. The range is 1-22 for zstd and 1-9 for gzip.

    NB2: The default program for gzip (at least with Debian) is the executable from zlib. With my workflows, libdeflate-gzip iscompatible and noticably faster.

    NB3: This benchmark is 2 years old. The latest releases of zstd are much better, see https://github.com/facebook/zstd/releases

    For a high compression, according to this benchmark xz can do slightly better, if you're willing to pay a 10ร— penalty on decompression.

    xz -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]

    zstd -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]

  • digital_video_introduction

    A hands-on introduction to video technology: image, video, codec (av1, vp9, h265) and more (ffmpeg encoding). Translations: ๐Ÿ‡บ๐Ÿ‡ธ ๐Ÿ‡จ๐Ÿ‡ณ ๐Ÿ‡ฏ๐Ÿ‡ต ๐Ÿ‡ฎ๐Ÿ‡น ๐Ÿ‡ฐ๐Ÿ‡ท ๐Ÿ‡ท๐Ÿ‡บ ๐Ÿ‡ง๐Ÿ‡ท ๐Ÿ‡ช๐Ÿ‡ธ

  • Project mention: Breakdown of AV1 Video Codec | news.ycombinator.com | 2023-12-25

    There's a great introduction to video tech, including codecs, at https://github.com/leandromoreira/digital_video_introduction

  • brotli

    Brotli compression format

  • Project mention: Jpegli: A New JPEG Coding Library | news.ycombinator.com | 2024-04-03

    JPEGLI = A small JPEG

    The suffix -li is used in Swiss German dialects. It forms a diminutive of the root word, by adding -li to the end of the root word to convey the smallness of the object and to convey a sense of intimacy or endearment.

    This obviously comes out of Google Zรผrich.

    Other notable Google projects using Swiss German:

    https://github.com/google/gipfeli high-speed compression

    Gipfeli = Croissant

    https://github.com/google/guetzli perceptual JPEG encoder

    Guetzli = Cookie

    https://github.com/weggli-rs/weggli semantic search tool

    Weggli = Bread roll

    https://github.com/google/brotli lossless compression

    Brรถtli = Small bread

  • PaddleNLP

    ๐Ÿ‘‘ Easy-to-use and powerful NLP and LLM library with ๐Ÿค— Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including ๐Ÿ—‚Text Classification, ๐Ÿ” Neural Search, โ“ Question Answering, โ„น๏ธ Information Extraction, ๐Ÿ“„ Document Intelligence, ๐Ÿ’Œ Sentiment Analysis etc.

  • BorgBackup

    Deduplicating archiver with compression and authenticated encryption.

  • Project mention: Ask HN: Open-source Windows 11 backup solutions | news.ycombinator.com | 2024-04-04

    i use - and recommend - "borgbackup": for example with the "vorta" graphical frontend

    * https://www.borgbackup.org/

    * https://vorta.borgbase.com/install/windows/

    just my 0.02โ‚ฌ

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo
  • JSZip

    Create, read and edit .zip files with Javascript

  • Project mention: Audio slicing with Javascript | /r/webdev | 2023-07-02

    You can use a lib named JSZip : https://stuk.github.io/jszip/ You can add binary data (your audio slice) inside a zip like that

  • LZ4

    Extremely Fast Compression algorithm

  • Project mention: Number sizes for LZ77 compression | /r/compression | 2023-04-30

    LZ4 is a bit more complicated, but seems faster: https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md

  • Compressor

    An android image compression library.

  • draco

    Draco is a library for compressing and decompressing 3D geometric meshes and point clouds. It is intended to improve the storage and transmission of 3D graphics.

  • Snappy

    A fast compressor/decompressor

  • pako

    high speed zlib port to javascript, works in browser & node.js

  • ZLib

    A massively spiffy yet delicately unobtrusive compression library.

  • Project mention: Zlib 1.3.1 Out | news.ycombinator.com | 2024-01-22
  • meshoptimizer

    Mesh optimization library that makes meshes smaller and faster to render

  • compress

    Optimized Go Compression Packages

  • Project mention: Chrome Feature: ZSTD Content-Encoding | news.ycombinator.com | 2024-04-01
  • CompactGUI

    Transparently compress active games and programs using Windows 10/11 APIs

  • Project mention: Is there a way to save space of ark? | /r/ARK | 2023-07-04

    CompactGUI

  • archiver

    Easily create & extract archives, and compress & decompress files of various formats

  • cute_headers

    Collection of cross-platform one-file C/C++ libraries with no dependencies, primarily used for games

  • SharpZipLib

    #ziplib is a Zip, GZip, Tar and BZip2 library written entirely in C# for the .NET platform.

  • PeaZip

    Free Zip / Unzip software and Rar file extractor. Cross-platform file and archive manager. Features volume spanning, compression, authenticated encryption. Supports 7Z, 7-Zip sfx, ACE, ARJ, Brotli, BZ2, CAB, CHM, CPIO, DEB, GZ, ISO, JAR, LHA/LZH, NSIS, OOo, PAQ/LPAQ, PEA, QUAD, RAR, RPM, split, TAR, Z, ZIP, ZIPX, Zstandard.

  • Project mention: RAR 7.0 Released | news.ycombinator.com | 2024-02-28

    Just saying that 7-Zip is great, both the compression (I use it for most of my data archival, because the higher compression ratio is worth the longer times to compress) and the software itself, pretty much what a good archival program on Windows should be like.

    On other OSes, I guess I just use whatever is included with the distro, or something like PeaZip: https://peazip.github.io/

    Curious to hear from RAR proponents.

  • Crunch

    Insane(ly slow but wicked good) PNG image optimization (by chrissimpkins)

  • zip.js

    JavaScript library to zip and unzip files supporting multi-core compression, compression streams, zip64, split files and encryption.

  • Project mention: Pigz: Parallel gzip for modern multi-processor, multi-core machines | news.ycombinator.com | 2023-05-12

    Similarly, if people are interested, I have coded the possibility to compress zip files on several cores in zip.js [1]. The approach is simpler as it consists of compressing the entries in parallel. It still offers a significant performance gain though when compressing multiple files in a zip file, which is often the nominal case.

    [1] https://github.com/gildas-lormeau/zip.js

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Compression related posts

Index

What are some of the best open-source Compression projects? This list will help you:

Project Stars
1 DeepSpeed 32,550
2 CyberChef 25,384
3 zstd 22,356
4 digital_video_introduction 15,095
5 brotli 13,111
6 PaddleNLP 11,386
7 BorgBackup 10,506
8 JSZip 9,463
9 LZ4 9,192
10 Compressor 6,992
11 draco 6,221
12 Snappy 5,977
13 pako 5,295
14 ZLib 5,264
15 meshoptimizer 4,959
16 compress 4,489
17 CompactGUI 4,491
18 archiver 4,228
19 cute_headers 4,094
20 SharpZipLib 3,604
21 PeaZip 3,572
22 Crunch 3,325
23 zip.js 3,273

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com