InfluxDB Platform is powered by columnar analytics, optimized for cost-efficient storage, and built with open data standards. Learn more →
Top 23 C++ Compression Projects
-
draco
Draco is a library for compressing and decompressing 3D geometric meshes and point clouds. It is intended to improve the storage and transmission of 3D graphics.
-
InfluxDB
Purpose built for real-time analytics at any scale. InfluxDB Platform is powered by columnar analytics, optimized for cost-efficient storage, and built with open data standards.
-
-
> Making a nanite mesh is complicated, with a lot of internal offsets for linking, and so far only Unreal Engine's editor does it.
meshoptimizer [1] is an OSS implementation of meshlet generation, which is what most people think of when they think of "Nanite's algorithm". Bevy, mentioned in a sibling reply, uses meshoptimizer as the generation tool.
(Strictly speaking, "Nanite" is a brand name that encompasses a large collection of techniques, including meshlets, software rasterization, streaming, etc. For clarity during technical discussions, I prefer to talk about individual techniques, since they're really separate, even though they complement one another. For example, software rasterization can be used without meshlets if your triangles are really small. Streaming can be useful even if you aren't using meshlets. And so on.)
[1]: https://github.com/zeux/meshoptimizer
-
caesium-image-compressor
Caesium is an image compression software that helps you store, send and share digital pictures, supporting JPG, PNG, WebP and TIFF formats. You can quickly reduce the file size (and resolution, if you want) by preserving the overall quality of the image.
I use this: https://saerasoft.com/caesium
-
-
Project mention: You can help Anna's Archive by seeding torrents | news.ycombinator.com | 2024-06-13
I wonder if there is any benefit to using something like DwarFS (https://github.com/mhx/dwarfs) for something like this.
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
Project mention: How do other motion capture suits manage large file sizes? | /r/gamedev | 2023-12-04
Cleaned and compressed are whatever they turn out to be. Use ACL, https://github.com/nfrechette/acl
-
compressonator
Tool suite for Texture and 3D Model Compression, Optimization and Analysis using CPUs, GPUs and APUs
-
Bareos
Bareos is a cross-network Open Source backup solution (licensed under AGPLv3) which preserves, archives, and recovers data from all major operating systems.
-
clp
Compressed Log Processor (CLP) is a free log management tool capable of compressing logs and searching the compressed logs without decompression. (by y-scope)
-
PGM-index
🏅State-of-the-art learned data structure that enables fast lookup, predecessor, range searches and updates in arrays of billions of items using orders of magnitude less space than traditional indexes
-
LZHAM
Lossless data compression codec with LZMA-like ratios but 1.5x-8x faster decompression speed, C/C++
-
-
SIMDCompressionAndIntersection
A C++ library to compress and intersect sorted lists of integers using SIMD instructions
-
GDCM
Grassroots DICOM read-only mirror. Only for Pull Request. Please report bug at http://sf.net/p/gdcm
-
zpaqfranz
Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
-
-
Project mention: An Empirical Evaluation of Columnar Storage Formats [pdf] | news.ycombinator.com | 2024-05-15
Lance v2 looks interesting. I like their meta-data + container story. Lacking SOTA encoding schemes though.
There is also Vortex (https://github.com/fulcrum-so/vortex). That has modern encoding schemes that we want to use.
BtrBlocks (https://github.com/maxi-k/btrblocks) from the Germans is another Parquet alternative.
Nimble (formerly Alpha) is a complicated story. We worked with the Velox team for over a year to open-source and extend it. But plans got stymied by legal. This was in collaboration with Meta + CWI + Nvidia + Voltron. We decided to a separate path because Nimble code has no spec/docs. Too tightly coupled with Velox/Folly.
Given that, we are working on a new file format. We hope to share our ideas/code later this year.
-
-
-
-
I meant the Kiwix dump (https://download.kiwix.org/zim/wikipedia_en_all_nopic.zim – careful, 60GB!).
At a first glance, the Wikimedia XML dump does not look substantially different from what Kiwix/ZIM does with compressed HTML: They're both compressed (bz2 for the Wikimedia dump, zstd or LZMA for Kiwix/ZIM), and both compress multiple files at once, so inter-file redundancy should hopefully be significantly reduced.
HTML seems a bit more verbose than the Mediawiki syntax (plus the XML header for each article), but I'd be surprised if that actually accounted for a 3x difference in size.
Then again, Kiwix seems to have experimented with shared dictionary brotli compression, which supposedly yields an >2x improvement: https://github.com/openzim/libzim/issues/144
I wonder if their current zstd implementation also uses shared dictionaries. If not, that might just be the reason: If ZIM compression chunks are much smaller than the bz2 streams of the Wikimedia dumps, there would still be a lot of redundancy between chunks.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
C++ Compression discussion
C++ Compression related posts
-
Another variable-length integer encoding
-
Show HN: Kanzi, fast lossless data compression
-
An Empirical Evaluation of Columnar Storage Formats [pdf]
-
GarlicOS lag
-
How do other motion capture suits manage large file sizes?
-
WikiReader
-
How are zlib, gzip and zip related?
-
A note from our sponsor - InfluxDB
www.influxdata.com | 15 Sep 2024
Index
What are some of the best open-source Compression projects in C++? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | draco | 6,452 |
| 2 | Snappy | 6,097 |
| 3 | meshoptimizer | 5,510 |
| 4 | caesium-image-compressor | 3,488 |
| 5 | bolt | 2,465 |
| 6 | dwarfs | 2,115 |
| 7 | dandere2x | 1,431 |
| 8 | acl | 1,337 |
| 9 | compressonator | 1,290 |
| 10 | Bareos | 969 |
| 11 | clp | 804 |
| 12 | PGM-index | 774 |
| 13 | LZHAM | 688 |
| 14 | EWAHBoolArray | 441 |
| 15 | SIMDCompressionAndIntersection | 417 |
| 16 | GDCM | 345 |
| 17 | zpaqfranz | 259 |
| 18 | grok | 225 |
| 19 | btrblocks | 213 |
| 20 | CIDLib | 208 |
| 21 | ancient | 208 |
| 22 | sevenzipjbinding | 174 |
| 23 | libzim | 165 |
Sponsored
Purpose built for real-time analytics at any scale.
InfluxDB Platform is powered by columnar analytics, optimized for cost-efficient storage, and built with open data standards.
www.influxdata.com