The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →
Top 23 Compression Open-Source Projects
-
DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
-
CyberChef
The Cyber Swiss Army Knife - a web app for encryption, encoding, compression and data analysis
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
digital_video_introduction
A hands-on introduction to video technology: image, video, codec (av1, vp9, h265) and more (ffmpeg encoding). Translations: 🇺🇸 🇨🇳 🇯🇵 🇮🇹 🇰🇷 🇷🇺 🇧🇷 🇪🇸
-
-
PaddleNLP
👑 Easy-to-use and powerful NLP and LLM library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis etc.
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
draco
Draco is a library for compressing and decompressing 3D geometric meshes and point clouds. It is intended to improve the storage and transmission of 3D graphics.
-
-
-
-
cute_headers
Collection of cross-platform one-file C/C++ libraries with no dependencies, primarily used for games
-
SharpZipLib
#ziplib is a Zip, GZip, Tar and BZip2 library written entirely in C# for the .NET platform.
-
PeaZip
Free Zip / Unzip software and Rar file extractor. Cross-platform file and archive manager. Features volume spanning, compression, authenticated encryption. Supports 7Z, 7-Zip sfx, ACE, ARJ, Brotli, BZ2, CAB, CHM, CPIO, DEB, GZ, ISO, JAR, LHA/LZH, NSIS, OOo, PAQ/LPAQ, PEA, QUAD, RAR, RPM, split, TAR, Z, ZIP, ZIPX, Zstandard.
-
zip.js
JavaScript library to zip and unzip files supporting multi-core compression, compression streams, zip64, split files and encryption.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: Can we discuss MLOps, Deployment, Optimizations, and Speed? | /r/LocalLLaMA | 2023-12-06DeepSpeed can handle parallelism concerns, and even offload data/model to RAM, or even NVMe (!?) . I'm surprised I don't see this project used more.
Then we take the encrypted text and use CyberChef to decrypt it.
Of course, you may get different results with another dataset.
gzip (zlib -6) [ratio=32%] [compr=35Mo/s] [dec=407Mo/s]
zstd (zstd -2) [ratio=32%] [compr=356Mo/s] [dec=1067Mo/s]
NB1: The default for zstd is -3, but the table only had -2. The difference is probably small. The range is 1-22 for zstd and 1-9 for gzip.
NB2: The default program for gzip (at least with Debian) is the executable from zlib. With my workflows, libdeflate-gzip iscompatible and noticably faster.
NB3: This benchmark is 2 years old. The latest releases of zstd are much better, see https://github.com/facebook/zstd/releases
For a high compression, according to this benchmark xz can do slightly better, if you're willing to pay a 10× penalty on decompression.
xz -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]
zstd -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]
There's a great introduction to video tech, including codecs, at https://github.com/leandromoreira/digital_video_introduction
Project mention: Node.js vs Angular: Navigating the Modern Web Development Landscape | dev.to | 2024-04-22Using tools like Brotli, you can boost your application’s load time. You can use the ngUpgrade library to mix AngularJS and Angular components to enhance runtime performance, bringing in hybrid applications that can be used with techniques like ahead-of-time (AOT) compilation, aiding in faster browser rendering.
Project mention: Ask HN: Open-source Windows 11 backup solutions | news.ycombinator.com | 2024-04-04i use - and recommend - "borgbackup": for example with the "vorta" graphical frontend
* https://www.borgbackup.org/
* https://vorta.borgbase.com/install/windows/
just my 0.02€
You can use a lib named JSZip : https://stuk.github.io/jszip/ You can add binary data (your audio slice) inside a zip like that
LZ4 is a bit more complicated, but seems faster: https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md
CompactGUI
Just saying that 7-Zip is great, both the compression (I use it for most of my data archival, because the higher compression ratio is worth the longer times to compress) and the software itself, pretty much what a good archival program on Windows should be like.
On other OSes, I guess I just use whatever is included with the distro, or something like PeaZip: https://peazip.github.io/
Curious to hear from RAR proponents.
Project mention: Pigz: Parallel gzip for modern multi-processor, multi-core machines | news.ycombinator.com | 2023-05-12Similarly, if people are interested, I have coded the possibility to compress zip files on several cores in zip.js [1]. The approach is simpler as it consists of compressing the entries in parallel. It still offers a significant performance gain though when compressing multiple files in a zip file, which is often the nominal case.
[1] https://github.com/gildas-lormeau/zip.js
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Compression related posts
- Crate Is Unmaintained
- Node.js vs Angular: Navigating the Modern Web Development Landscape
- XZ backdoor story – Initial analysis
- Xz: Update maintainer and author info. The other maintainer suddenly disappeared
- Ratarmount: Access large archives as a filesystem efficiently
- Bzip2 Format Specification [pdf]
- Thanks Andres Freud
-
A note from our sponsor - WorkOS
workos.com | 25 Apr 2024
Index
What are some of the best open-source Compression projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | DeepSpeed | 32,550 |
| 2 | CyberChef | 25,541 |
| 3 | zstd | 22,356 |
| 4 | digital_video_introduction | 15,095 |
| 5 | brotli | 13,123 |
| 6 | PaddleNLP | 11,386 |
| 7 | BorgBackup | 10,526 |
| 8 | JSZip | 9,482 |
| 9 | LZ4 | 9,208 |
| 10 | Compressor | 6,992 |
| 11 | draco | 6,251 |
| 12 | Snappy | 5,987 |
| 13 | pako | 5,295 |
| 14 | ZLib | 5,278 |
| 15 | meshoptimizer | 4,969 |
| 16 | CompactGUI | 4,491 |
| 17 | compress | 4,489 |
| 18 | archiver | 4,233 |
| 19 | cute_headers | 4,105 |
| 20 | SharpZipLib | 3,608 |
| 21 | PeaZip | 3,572 |
| 22 | Crunch | 3,325 |
| 23 | zip.js | 3,278 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com