Appwrite is an open source backend server that helps you build native iOS applications much faster with realtime APIs for authentication, databases, files storage, cloud functions and much more! Learn more →
Top 23 Compression Open-Source Projects
-
CyberChef
The Cyber Swiss Army Knife - a web app for encryption, encoding, compression and data analysis
CyberChef https://gchq.github.io/CyberChef/
-
Project mention: Proxmox Backup Server with tape drive - How to determine datastore size? | reddit.com/r/Proxmox | 2023-02-13
Look at the "Benchmarks" table. https://github.com/facebook/zstd/blob/dev/README.md
-
Appwrite
Appwrite - The Open Source Firebase alternative introduces iOS support . Appwrite is an open source backend server that helps you build native iOS applications much faster with realtime APIs for authentication, databases, files storage, cloud functions and much more!
-
digital_video_introduction
A hands-on introduction to video technology: image, video, codec (av1, vp9, h265) and more (ffmpeg encoding). Translations: 🇺🇸 🇨🇳 🇯🇵 🇮🇹 🇰🇷 🇷🇺
A couple of AV centric scenarios I'd explore: - Set up two speakers fed with a mono signal, flip the phase on one and have students move around the area. From this introduce constructive and destructive interference and some basic acoustics. - Using an oscilloscope (or just prepped graphics) show a sine wave on an unbalanced line, how this is paired with an inverted copy of itself on a balanced line, and how this works to cancel transmission noise. - With an analogue signal now pictured, introduce analogue to digital conversion and what sample rate and bit depth mean. From this you can also start to introduce why gain staging is important in both pure analogue systems and those where you're bouncing between analogue and digital domains. - Introduce video: colour channels, different ways of representing colour and some (very surface level) compression basics like chroma subsampling. Really good resource for that here. - Introduce base patterns for getting any signals arounds: dedicated circuits (analogue audio, HDMI video, RS-232 control, logic IO) and network based (again for audio, video, and control). Dive intro signal flow, how to read a schematic and where reasoning about each of these modes differs.
-
We serve your site from a global cache location close to your visitors to make sure your site loads fast. In addition, we use an advanced HTML and text compression algorithm called Brotli. Compressed content is now cached, so we can send it directly to your visitors instead of compressing each request individually. In our tests this often improves loading speed by up to 2x, which will have a very positive impact on your Lighthouse scores like LCP. This will be especially noticeable on larger sites, so you can scale your site without worry.
-
DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Project mention: Accelerate config options meaning and which should i choose | reddit.com/r/StableDiffusion | 2023-03-14 -
Project mention: Making separate partitions for /boot and /home | reddit.com/r/linuxquestions | 2023-03-15
Borg. If desired, in combination with Vorta or Pika.
-
Project mention: A Website to Archive Moodle Course Files - My first React Project | reddit.com/r/reactjs | 2023-02-27
The ZIP file functionality was implemented using jszip - I'd originally wanted to download the files directly to a folder on the user's filesystem (coming from a scripting background) but turns out unrestricted filesystem access like that isn't something that browsers can do (at least from my limited research).
-
Sonar
Write Clean JavaScript Code. Always.. Sonar helps you commit clean code every time. With over 300 unique rules to find JavaScript bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.
-
According to these https://github.com/lz4/lz4 values you need around ten (10) quite modern cores in parallel to accomplish around 8GB/s.
-
Project mention: setResult and finish don't work inside any Coroutine Scope. Possible reasons? | reddit.com/r/androiddev | 2022-07-14
We use the Compressor library in our app. I upgraded it to V3 today, and that requires that the compression code be run inside a Coroutine Scope.
-
Project mention: Why I enjoy using the Nim programming language at Reddit. | reddit.com/r/RedditEng | 2022-11-14
Another example of Nim being really fast is the supersnappy library. This library benchmarks faster than Google’s C or C++ Snappy implementation.
-
draco
Draco is a library for compressing and decompressing 3D geometric meshes and point clouds. It is intended to improve the storage and transmission of 3D graphics.
Project mention: how can I reduce size of a gltf file for web use? | reddit.com/r/Frontend | 2022-08-15 -
try pako
-
Seeing the text in red got me thinking for a moment, "wow, didn't realize pirates had such love for an open-source compression library"
-
Project mention: [Part 7] Update of my Vulkan renderer: LODs, Multiple different meshes, Memory Allocator, Render architecture and more | reddit.com/r/vulkan | 2022-10-08
Implemented Mesh LOD system. LOD selection is done completely on GPU. I generate LODs using meshoptimizer.
-
Project mention: Ouch - simple compression and decompression for your terminal | reddit.com/r/commandline | 2023-01-07
archiver is also a very cool tool. I think its cli is somewhat similar to this.
-
cute_headers
Collection of cross-platform one-file C/C++ libraries with no dependencies, primarily used for games
Project mention: How many colors are too many colors for Windows Terminal? | news.ycombinator.com | 2022-05-14- https://github.com/RandyGaul/cute_headers/blob/master/cute_s...
It's a simple and relatively straightforward approach that a sufficiently bright programmer would come up in their own while looking at the design constraints though, so overall I find it a bit meaningless to find the ultimate person for the "original idea".
-
It's no surprise that "fast" Go libraries are actually just assembly: https://github.com/klauspost/compress/blob/master/zstd/seqdec_amd64.s (just one file out of several, for just one architecture, for just one compression algorithm!)
-
Project mention: Why is Tekken 7's size 2x that of Elden Ring on my PC. | reddit.com/r/Tekken | 2023-02-27
After you delete the cinematics you can compress the game folder with: https://github.com/IridiumIO/CompactGUI to save another 10-15gb
-
SharpZipLib
#ziplib is a Zip, GZip, Tar and BZip2 library written entirely in C# for the .NET platform.
Project mention: Best practice to generate ZIP file for client side download? | reddit.com/r/Blazor | 2022-08-01public class DownloadStreamedZippedFilesActionResult : FileResult { private const int Zip64CentralDirectoryRecordSize = 56; private const int Zip64CentralDirectoryLocatorSize = 20; /// /// This was obtained by debugging DotNetZip. It might change in newer versions of the library. The size is hardcoded with no constant available, not even a private one. /// private const int DotNetZipLocalFileHeaderExtraFieldLength = 20; private const int DotNetZipCentralDirectoryHeaderExtraFieldLength = 32; private const string ZipExtension = ".zip"; private static readonly ILogger Logger = ApplicationLogging.CreateLogger(); private readonly IReadOnlyCollection files; private readonly CancellationToken cancellationToken; public DownloadStreamedZippedFilesActionResult(string fileName, IEnumerable files, CancellationToken cancellationToken) : base(MediaTypeConverter.GetMimeType(ZipExtension)) { this.files = files.ToArray(); FileDownloadName = fileName + ZipExtension; this.cancellationToken = cancellationToken; } public DownloadStreamedZippedFilesActionResult(string fileName, IEnumerable files) : this(fileName, files, new CancellationToken(false)) { } public override Task ExecuteResultAsync(ActionContext context) { var response = context.HttpContext.Response; response.OnStarting(() => { var contentDisposition = new ContentDispositionHeaderValue("attachment"); contentDisposition.SetHttpFileName(FileDownloadName); response.Headers.Add("access-control-expose-headers", new[] { "content-disposition", "content-length" }); response.Headers.Add("content-disposition", contentDisposition.ToString()); response.Headers.ContentLength = CalculateContentLength(); return Task.FromResult(0); }); var callback = GetCallback(); return callback(response.Body); } private Func GetCallback() { return async (outputStream) => { byte[] buffer = new byte[4096]; try { var countingStream = new CountingStream(outputStream); using (outputStream) using (var zipStream = new Ionic.Zip.ZipOutputStream(countingStream, true)) { zipStream.CompressionLevel = Ionic.Zlib.CompressionLevel.Level0; zipStream.CompressionMethod = Ionic.Zip.CompressionMethod.None; zipStream.EnableZip64 = Zip64Option.Always; foreach (var file in files) { using (var fileContent = await file.GetContentAsync()) { zipStream.PutNextEntry(file.Name); StreamUtils.Copy(fileContent, zipStream, buffer); zipStream.Flush(); outputStream.Flush(); } cancellationToken.ThrowIfCancellationRequested(); } zipStream.Flush(); zipStream.Close(); outputStream.Flush(); } } catch (Exception e) { Logger.LogError(e, "Unexpected error streaming {FileName}", FileDownloadName); throw; } }; } private long CalculateContentLength() { /* ZIP file size constants according to the specification https://users.cs.jmu.edu/buchhofp/forensics/formats/pkzip.html and https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT * Please note this only works with compression type 0 (store, no compression). * This formula also takes into account the additional content DotNetZip adds in the extra fields. It might be different for other zip providers. * ZIP constants come from SharpZipLib https://github.com/icsharpcode/SharpZipLib/blob/master/src/ICSharpCode.SharpZipLib/Zip/ZipConstants.cs */ return files.Count * (ZipConstants.LocalHeaderBaseSize + ZipConstants.Zip64DataDescriptorSize + ZipConstants.CentralHeaderBaseSize + DotNetZipLocalFileHeaderExtraFieldLength + DotNetZipCentralDirectoryHeaderExtraFieldLength) + 2 * files.Sum(f => f.Name.Length) + files.Sum(f => f.Size) + ZipConstants.EndOfCentralRecordBaseSize + Zip64CentralDirectoryRecordSize + Zip64CentralDirectoryLocatorSize; } } public record class StreamedFile { private readonly Func> fileFunc; public StreamedFile(string name, long size, Func> fileFunc) { this.fileFunc = fileFunc; Name = name; Size = size; } public string Name { get; } public long Size { get; } /// /// Callback to get the file from S3, Azure Storage or wherever /// public Task GetContentAsync() => fileFunc(); }
-
Project mention: (Newbie) How can I replicate this UI? Would SpriteKit be the best option? | reddit.com/r/iOSProgramming | 2023-03-08
If you go the png/jpg route, don’t forget to crunch your assets to save disk space.
-
Does anybody know of a steganography app that can be containerized in docker similar to stegcloak or PixelKnot? really would like a selfhosted option for these apps.
-
zip.js
JavaScript library to zip and unzip files supporting multi-core compression, compression streams, zip64, split files and encryption.
-
PeaZip
Free Zip / Unzip software and Rar file extractor. Cross-platform file and archive manager. Features volume spanning, compression, authenticated encryption. Supports 7Z, 7-Zip sfx, ACE, ARJ, Brotli, BZ2, CAB, CHM, CPIO, DEB, GZ, ISO, JAR, LHA/LZH, NSIS, OOo, PAQ/LPAQ, PEA, QUAD, RAR, RPM, split, TAR, Z, ZIP, ZIPX, Zstandard.
Hey… put the lamp back down, we don’t gotta throw light fixtures to get what we want. Take a look at PeaZip instead. It’s a free, open source, cross platform file archiver! Peazip is totally free and it’s available for Linux, macOS and Windows! They even have super nerdy BSD and CLI versions for you hardcore nerds out there. PeaZip can extract just about any compression format you could want … yeah even ARC. PeaZip’s even kicked security up a notch! They’ve included a sweet “2fa” mechanism for your most prized encrypted archives. So go ahead and dump that way out of date trial copy of WinRAR from 2017 and download PeaZip.
-
InfluxDB
Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2023-03-21.
Compression related posts
- Rsyncing 20TB locally
- Dear Pirates Donate as much as you can
- Which is better Opus or AC3?
- Compress a file using Brotli client side
- Daily General Discussion - March 16, 2023
- Making separate partitions for /boot and /home
- Simple, fast and safety alternative for unzip
-
A note from our sponsor - Appwrite
appwrite.io | 21 Mar 2023
Index
What are some of the best open-source Compression projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | CyberChef | 20,527 |
| 2 | zstd | 19,774 |
| 3 | digital_video_introduction | 13,426 |
| 4 | brotli | 11,965 |
| 5 | DeepSpeed | 9,536 |
| 6 | BorgBackup | 9,221 |
| 7 | JSZip | 8,725 |
| 8 | LZ4 | 8,041 |
| 9 | Compressor | 6,738 |
| 10 | Snappy | 5,567 |
| 11 | draco | 5,524 |
| 12 | pako | 4,729 |
| 13 | ZLib | 4,248 |
| 14 | meshoptimizer | 4,017 |
| 15 | archiver | 3,907 |
| 16 | cute_headers | 3,740 |
| 17 | compress | 3,561 |
| 18 | CompactGUI | 3,382 |
| 19 | SharpZipLib | 3,305 |
| 20 | Crunch | 3,233 |
| 21 | StegCloak | 2,870 |
| 22 | zip.js | 2,839 |
| 23 | PeaZip | 2,647 |
Access the most powerful time series database as a service
Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
www.influxdata.com