SaaSHub helps you find the best software and product alternatives Learn more →
Top 23 Python Compression Projects
-
DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Project mention: Can we discuss MLOps, Deployment, Optimizations, and Speed? | /r/LocalLLaMA | 2023-12-06DeepSpeed can handle parallelism concerns, and even offload data/model to RAM, or even NVMe (!?) . I'm surprised I don't see this project used more.
-
Scout Monitoring
Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
-
PaddleNLP
👑 Easy-to-use and powerful NLP and LLM library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis etc.
-
Project mention: Ask HN: How do you manage files and backups as an individual? | news.ycombinator.com | 2024-07-14
I started using Nextcloud for file/contact/calendar syncing a few years, and have gradually moved most of my digital life into it. Documents and photos, but also scripts to automatically set some things up for me when I do a fresh install of Linux (I've been playing around with a few different distros lately). The only thing that doesn't live in Nextcloud are some old DVD rips, and that's mostly due to "haven't gotten around to it yet". Besides those, if it's not in Nextcloud, it's not something I care too much about losing if a disk were to fail.
My Nextcloud instance is then backed up to another drive on the same machine plus two off-site locations - an old server I still run at my parents' house, and and external HDD my friend let me plug in to his server. The on- and off-site backups are done using `borg` (https://www.borgbackup.org/), which does reduplication and encryption (with the keys backed up in 1Password).
I've been meaning to set up an automated restore on one of the offsite servers - a script to automatically unpack the latest backup, set up a fresh DB, and make things read-only - firstly to verify that the backups are functional and complete, but also as a backup in case my home server or home internet goes down. I know in _theory_ I've got everything I need in the backups to do a full restore, but I can't recall the last time I actually tried it out...
-
-
Project mention: Veles: Open-source tool for binary data analysis | news.ycombinator.com | 2024-07-25
-
aimet
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
-
Project mention: Syncthing – A decentralized continuous file synchronization program | news.ycombinator.com | 2024-08-18
You could use Syncthing just to empty the incoming files from your phone (ingest) and then move the photos via cron to a second folder (also Syncthing) which is just shared with the replicas.
Another approach would be to push the files from Syncthing to borg (borgmatic can do replicas) https://torsion.org/borgmatic/
-
InfluxDB
Purpose built for real-time analytics at any scale. InfluxDB Platform is powered by columnar analytics, optimized for cost-efficient storage, and built with open data standards.
-
model-optimization
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
-
gan-compression
[CVPR 2020] GAN Compression: Efficient Architectures for Interactive Conditional GANs
One elegant approach for this I’ve found is this https://github.com/mit-han-lab/gan-compression They basically train an “all in one” network from which you can extract small or large models afterwards (with optional additional finetuning to improve the selected channel size combinations)
-
-
-
-
ratarmount
Access large archives as a filesystem efficiently, e.g., TAR, RAR, ZIP, GZ, BZ2, XZ, ZSTD archives
Project mention: Ask HN: A better Criu Alternative for decompression software / Erlang? | news.ycombinator.com | 2024-09-15So Criu (https://criu.org/) , is a checkpoint restore utility which could be used with docker.
Now the main use case of this is docker live migration from which I learned something
But I also realised that this allows (atleast on linux) (though you could use wsl on windows) to allow decompression software like winrar , peazip etc. or cli tools like unrar-free to shutdown while the decompression is taking place and also erlang + criu just feels like a very awesome combination
Now we could do something like this by taking random access (https://github.com/mxmlnkn/ratarmount)
but I was wondering about this if we could have some sort of backup software where it would continue downloading the file right from where its left even the computer shuts down.
and though there is dmtcp , I am not exactly sure if that's the right cause since on criu's website comparing this.
It says "Such behavior is very dangerous, as application might see wrong files in the /proc filesystem if it will try to access one via its PID."
Now I was wondering if we could have a general sort of Virtual Machine which could run a specific language (especially the erlang , especially erlang with its let it fail approach could be nice) / For decompression software as well.
Is criu the best we can get within the linux world , has somebody thought of creating a very lightweight os meant to be as a virtual machine with checkpoint / restore.
Or could we rather instead of Criu , actually try to do this without Criu , right in the decompression software tooling ?
I am somewhat confused about this , and I would really like it if someone with experience could simplify where I should try to put my efforts into doing such project
-
-
swin2sr
[ECCV] Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration. Advances in Image Manipulation (AIM) workshop ECCV 2022. Try it out! over 3.3M runs https://replicate.com/mv-lab/swin2sr
-
-
llm-compressor
Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM
Project mention: LLM compressor: compress models for efficient deployment | news.ycombinator.com | 2024-08-20 -
-
SecretPixel
SecretPixel is a cutting-edge steganography tool designed to securely conceal sensitive information within images. It stands out in the realm of digital steganography by combining advanced encryption, compression, and a seeded Least Significant Bit (LSB) technique to provide a robust solution for embedding data undetectably.
Project mention: Show HN: SecretPixel – Advanced Image Steganography Tool | news.ycombinator.com | 2024-02-03I've tested Binwalk on all the example files, and the BMP and TGA samples didn't show any zlib compressed data (https://i.ibb.co/R4WWsV2/2024-02-04-16h22-31.png). Can you please confirm that you have used the files from the example folder (https://github.com/x011/SecretPixel/tree/main/examples)? I will try to work on the PNG and TIFF cases soon. Thanks for the paper :)
-
-
-
-
Project mention: Show HN: Georaptor – A Python Geohash Compression Tool v3.0.0 | news.ycombinator.com | 2024-04-03
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python Compression discussion
Python Compression related posts
-
Ask HN: A better Criu Alternative for decompression software / Erlang?
-
LLM compressor: compress models for efficient deployment
-
Ask HN: How do you manage files and backups as an individual?
-
Ratarmount: Access large archives as a filesystem efficiently
-
TSAC: Low Bitrate Audio Compression
-
Show HN: SecretPixel – Advanced Image Steganography Tool
-
What do you use for VPS backup? Would improved borg setup - pull mode - be enough? Or, do you use something else?
-
A note from our sponsor - SaaSHub
www.saashub.com | 20 Sep 2024
Index
What are some of the best open-source Compression projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | DeepSpeed | 34,711 |
| 2 | PaddleNLP | 11,958 |
| 3 | BorgBackup | 10,962 |
| 4 | Crunch | 3,352 |
| 5 | unblob | 2,145 |
| 6 | aimet | 2,087 |
| 7 | borgmatic | 1,730 |
| 8 | model-optimization | 1,487 |
| 9 | gan-compression | 1,103 |
| 10 | Awesome-Efficient-LLM | 1,110 |
| 11 | nncf | 910 |
| 12 | compression | 850 |
| 13 | ratarmount | 700 |
| 14 | refinery | 627 |
| 15 | swin2sr | 576 |
| 16 | zipfly | 522 |
| 17 | llm-compressor | 411 |
| 18 | pythonlibs | 350 |
| 19 | SecretPixel | 306 |
| 20 | KVQuant | 280 |
| 21 | 3d-model-convert-to-gltf | 247 |
| 22 | DictDataBase | 231 |
| 23 | georaptor | 188 |
Sponsored
Free Django app performance insights with Scout Monitoring
Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
www.scoutapm.com