Compression

Top 23 Compression Open-Source Projects

  • CyberChef

    The Cyber Swiss Army Knife - a web app for encryption, encoding, compression and data analysis

    Project mention: Compress images - selfhosted solution | reddit.com/r/selfhosted | 2023-03-15

    CyberChef https://gchq.github.io/CyberChef/

  • zstd

    Zstandard - Fast real-time compression algorithm

    Project mention: Proxmox Backup Server with tape drive - How to determine datastore size? | reddit.com/r/Proxmox | 2023-02-13

    Look at the "Benchmarks" table. https://github.com/facebook/zstd/blob/dev/README.md

  • Appwrite

    Appwrite - The Open Source Firebase alternative introduces iOS support . Appwrite is an open source backend server that helps you build native iOS applications much faster with realtime APIs for authentication, databases, files storage, cloud functions and much more!

  • digital_video_introduction

    A hands-on introduction to video technology: image, video, codec (av1, vp9, h265) and more (ffmpeg encoding). Translations: 🇺🇸 🇨🇳 🇯🇵 🇮🇹 🇰🇷 🇷🇺

    Project mention: Topics to cover in a 101 AV class | reddit.com/r/CommercialAV | 2023-02-02

    A couple of AV centric scenarios I'd explore: - Set up two speakers fed with a mono signal, flip the phase on one and have students move around the area. From this introduce constructive and destructive interference and some basic acoustics. - Using an oscilloscope (or just prepped graphics) show a sine wave on an unbalanced line, how this is paired with an inverted copy of itself on a balanced line, and how this works to cancel transmission noise. - With an analogue signal now pictured, introduce analogue to digital conversion and what sample rate and bit depth mean. From this you can also start to introduce why gain staging is important in both pure analogue systems and those where you're bouncing between analogue and digital domains. - Introduce video: colour channels, different ways of representing colour and some (very surface level) compression basics like chroma subsampling. Really good resource for that here. - Introduce base patterns for getting any signals arounds: dedicated circuits (analogue audio, HDMI video, RS-232 control, logic IO) and network based (again for audio, video, and control). Dive intro signal flow, how to read a schematic and where reasoning about each of these modes differs.

  • brotli

    Brotli compression format

    Project mention: Framer Update: 2x Faster Sites | reddit.com/r/framer | 2023-02-02

    We serve your site from a global cache location close to your visitors to make sure your site loads fast. In addition, we use an advanced HTML and text compression algorithm called Brotli. Compressed content is now cached, so we can send it directly to your visitors instead of compressing each request individually. In our tests this often improves loading speed by up to 2x, which will have a very positive impact on your Lighthouse scores like LCP. This will be especially noticeable on larger sites, so you can scale your site without worry.

  • DeepSpeed

    DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

    Project mention: Accelerate config options meaning and which should i choose | reddit.com/r/StableDiffusion | 2023-03-14
  • BorgBackup

    Deduplicating archiver with compression and authenticated encryption.

    Project mention: Making separate partitions for /boot and /home | reddit.com/r/linuxquestions | 2023-03-15

    Borg. If desired, in combination with Vorta or Pika.

  • JSZip

    Create, read and edit .zip files with Javascript

    Project mention: A Website to Archive Moodle Course Files - My first React Project | reddit.com/r/reactjs | 2023-02-27

    The ZIP file functionality was implemented using jszip - I'd originally wanted to download the files directly to a folder on the user's filesystem (coming from a scripting background) but turns out unrestricted filesystem access like that isn't something that browsers can do (at least from my limited research).

  • Sonar

    Write Clean JavaScript Code. Always.. Sonar helps you commit clean code every time. With over 300 unique rules to find JavaScript bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  • LZ4

    Extremely Fast Compression algorithm

    Project mention: Rsyncing 20TB locally | reddit.com/r/zfs | 2023-03-21

    According to these https://github.com/lz4/lz4 values you need around ten (10) quite modern cores in parallel to accomplish around 8GB/s.

  • Compressor

    An android image compression library.

    Project mention: setResult and finish don't work inside any Coroutine Scope. Possible reasons? | reddit.com/r/androiddev | 2022-07-14

    We use the Compressor library in our app. I upgraded it to V3 today, and that requires that the compression code be run inside a Coroutine Scope.

  • Snappy

    A fast compressor/decompressor

    Project mention: Why I enjoy using the Nim programming language at Reddit. | reddit.com/r/RedditEng | 2022-11-14

    Another example of Nim being really fast is the supersnappy library. This library benchmarks faster than Google’s C or C++ Snappy implementation.

  • draco

    Draco is a library for compressing and decompressing 3D geometric meshes and point clouds. It is intended to improve the storage and transmission of 3D graphics.

    Project mention: how can I reduce size of a gltf file for web use? | reddit.com/r/Frontend | 2022-08-15
  • pako

    high speed zlib port to javascript, works in browser & node.js

    Project mention: Gzip library for client-side compression | reddit.com/r/webdev | 2022-06-30

    try pako

  • ZLib

    A massively spiffy yet delicately unobtrusive compression library.

    Project mention: Dear Pirates Donate as much as you can | reddit.com/r/Piracy | 2023-03-21

    Seeing the text in red got me thinking for a moment, "wow, didn't realize pirates had such love for an open-source compression library"

  • meshoptimizer

    Mesh optimization library that makes meshes smaller and faster to render

    Project mention: [Part 7] Update of my Vulkan renderer: LODs, Multiple different meshes, Memory Allocator, Render architecture and more | reddit.com/r/vulkan | 2022-10-08

    Implemented Mesh LOD system. LOD selection is done completely on GPU. I generate LODs using meshoptimizer.

  • archiver

    Easily create & extract archives, and compress & decompress files of various formats

    Project mention: Ouch - simple compression and decompression for your terminal | reddit.com/r/commandline | 2023-01-07

    archiver is also a very cool tool. I think its cli is somewhat similar to this.

  • cute_headers

    Collection of cross-platform one-file C/C++ libraries with no dependencies, primarily used for games

    Project mention: How many colors are too many colors for Windows Terminal? | news.ycombinator.com | 2022-05-14

    - https://github.com/RandyGaul/cute_headers/blob/master/cute_s...

    It's a simple and relatively straightforward approach that a sufficiently bright programmer would come up in their own while looking at the design constraints though, so overall I find it a bit meaningless to find the ultimate person for the "original idea".

  • compress

    Optimized Go Compression Packages

    Project mention: When is go not a good choice? | reddit.com/r/golang | 2023-03-03

    It's no surprise that "fast" Go libraries are actually just assembly: https://github.com/klauspost/compress/blob/master/zstd/seqdec_amd64.s (just one file out of several, for just one architecture, for just one compression algorithm!)

  • CompactGUI

    Transparently compress active games and programs using Windows 10/11 APIs

    Project mention: Why is Tekken 7's size 2x that of Elden Ring on my PC. | reddit.com/r/Tekken | 2023-02-27

    After you delete the cinematics you can compress the game folder with: https://github.com/IridiumIO/CompactGUI to save another 10-15gb

  • SharpZipLib

    #ziplib is a Zip, GZip, Tar and BZip2 library written entirely in C# for the .NET platform.

    Project mention: Best practice to generate ZIP file for client side download? | reddit.com/r/Blazor | 2022-08-01

    public class DownloadStreamedZippedFilesActionResult : FileResult { private const int Zip64CentralDirectoryRecordSize = 56; private const int Zip64CentralDirectoryLocatorSize = 20; /// /// This was obtained by debugging DotNetZip. It might change in newer versions of the library. The size is hardcoded with no constant available, not even a private one. /// private const int DotNetZipLocalFileHeaderExtraFieldLength = 20; private const int DotNetZipCentralDirectoryHeaderExtraFieldLength = 32; private const string ZipExtension = ".zip"; private static readonly ILogger Logger = ApplicationLogging.CreateLogger(); private readonly IReadOnlyCollection files; private readonly CancellationToken cancellationToken; public DownloadStreamedZippedFilesActionResult(string fileName, IEnumerable files, CancellationToken cancellationToken) : base(MediaTypeConverter.GetMimeType(ZipExtension)) { this.files = files.ToArray(); FileDownloadName = fileName + ZipExtension; this.cancellationToken = cancellationToken; } public DownloadStreamedZippedFilesActionResult(string fileName, IEnumerable files) : this(fileName, files, new CancellationToken(false)) { } public override Task ExecuteResultAsync(ActionContext context) { var response = context.HttpContext.Response; response.OnStarting(() => { var contentDisposition = new ContentDispositionHeaderValue("attachment"); contentDisposition.SetHttpFileName(FileDownloadName); response.Headers.Add("access-control-expose-headers", new[] { "content-disposition", "content-length" }); response.Headers.Add("content-disposition", contentDisposition.ToString()); response.Headers.ContentLength = CalculateContentLength(); return Task.FromResult(0); }); var callback = GetCallback(); return callback(response.Body); } private Func GetCallback() { return async (outputStream) => { byte[] buffer = new byte[4096]; try { var countingStream = new CountingStream(outputStream); using (outputStream) using (var zipStream = new Ionic.Zip.ZipOutputStream(countingStream, true)) { zipStream.CompressionLevel = Ionic.Zlib.CompressionLevel.Level0; zipStream.CompressionMethod = Ionic.Zip.CompressionMethod.None; zipStream.EnableZip64 = Zip64Option.Always; foreach (var file in files) { using (var fileContent = await file.GetContentAsync()) { zipStream.PutNextEntry(file.Name); StreamUtils.Copy(fileContent, zipStream, buffer); zipStream.Flush(); outputStream.Flush(); } cancellationToken.ThrowIfCancellationRequested(); } zipStream.Flush(); zipStream.Close(); outputStream.Flush(); } } catch (Exception e) { Logger.LogError(e, "Unexpected error streaming {FileName}", FileDownloadName); throw; } }; } private long CalculateContentLength() { /* ZIP file size constants according to the specification https://users.cs.jmu.edu/buchhofp/forensics/formats/pkzip.html and https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT * Please note this only works with compression type 0 (store, no compression). * This formula also takes into account the additional content DotNetZip adds in the extra fields. It might be different for other zip providers. * ZIP constants come from SharpZipLib https://github.com/icsharpcode/SharpZipLib/blob/master/src/ICSharpCode.SharpZipLib/Zip/ZipConstants.cs */ return files.Count * (ZipConstants.LocalHeaderBaseSize + ZipConstants.Zip64DataDescriptorSize + ZipConstants.CentralHeaderBaseSize + DotNetZipLocalFileHeaderExtraFieldLength + DotNetZipCentralDirectoryHeaderExtraFieldLength) + 2 * files.Sum(f => f.Name.Length) + files.Sum(f => f.Size) + ZipConstants.EndOfCentralRecordBaseSize + Zip64CentralDirectoryRecordSize + Zip64CentralDirectoryLocatorSize; } } public record class StreamedFile { private readonly Func> fileFunc; public StreamedFile(string name, long size, Func> fileFunc) { this.fileFunc = fileFunc; Name = name; Size = size; } public string Name { get; } public long Size { get; } /// /// Callback to get the file from S3, Azure Storage or wherever /// public Task GetContentAsync() => fileFunc(); }

  • Crunch

    Insane(ly slow but wicked good) PNG image optimization (by chrissimpkins)

    Project mention: (Newbie) How can I replicate this UI? Would SpriteKit be the best option? | reddit.com/r/iOSProgramming | 2023-03-08

    If you go the png/jpg route, don’t forget to crunch your assets to save disk space.

  • StegCloak

    Hide secrets with invisible characters in plain text securely using passwords 🧙🏻‍♂️⭐

    Project mention: Docker options for stegangraphy? | reddit.com/r/selfhosted | 2023-02-11

    Does anybody know of a steganography app that can be containerized in docker similar to stegcloak or PixelKnot? really would like a selfhosted option for these apps.

  • zip.js

    JavaScript library to zip and unzip files supporting multi-core compression, compression streams, zip64, split files and encryption.

  • PeaZip

    Free Zip / Unzip software and Rar file extractor. Cross-platform file and archive manager. Features volume spanning, compression, authenticated encryption. Supports 7Z, 7-Zip sfx, ACE, ARJ, Brotli, BZ2, CAB, CHM, CPIO, DEB, GZ, ISO, JAR, LHA/LZH, NSIS, OOo, PAQ/LPAQ, PEA, QUAD, RAR, RPM, split, TAR, Z, ZIP, ZIPX, Zstandard.

    Project mention: Daily General Discussion - March 16, 2023 | reddit.com/r/ethfinance | 2023-03-16

    Hey… put the lamp back down, we don’t gotta throw light fixtures to get what we want. Take a look at PeaZip instead. It’s a free, open source, cross platform file archiver! Peazip is totally free and it’s available for Linux, macOS and Windows! They even have super nerdy BSD and CLI versions for you hardcore nerds out there. PeaZip can extract just about any compression format you could want … yeah even ARC. PeaZip’s even kicked security up a notch! They’ve included a sweet “2fa” mechanism for your most prized encrypted archives. So go ahead and dump that way out of date trial copy of WinRAR from 2017 and download PeaZip.

  • InfluxDB

    Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-03-21.

Compression related posts

Index

What are some of the best open-source Compression projects? This list will help you:

Project Stars
1 CyberChef 20,527
2 zstd 19,774
3 digital_video_introduction 13,426
4 brotli 11,965
5 DeepSpeed 9,536
6 BorgBackup 9,221
7 JSZip 8,725
8 LZ4 8,041
9 Compressor 6,738
10 Snappy 5,567
11 draco 5,524
12 pako 4,729
13 ZLib 4,248
14 meshoptimizer 4,017
15 archiver 3,907
16 cute_headers 3,740
17 compress 3,561
18 CompactGUI 3,382
19 SharpZipLib 3,305
20 Crunch 3,233
21 StegCloak 2,870
22 zip.js 2,839
23 PeaZip 2,647
Access the most powerful time series database as a service
Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
www.influxdata.com