zstd
rsync
Our great sponsors
zstd | rsync | |
---|---|---|
105 | 39 | |
22,356 | 2,295 | |
2.2% | 8.8% | |
9.7 | 3.5 | |
3 days ago | 9 days ago | |
C | C | |
GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
zstd
-
Chrome Feature: ZSTD Content-Encoding
Of course, you may get different results with another dataset.
gzip (zlib -6) [ratio=32%] [compr=35Mo/s] [dec=407Mo/s]
zstd (zstd -2) [ratio=32%] [compr=356Mo/s] [dec=1067Mo/s]
NB1: The default for zstd is -3, but the table only had -2. The difference is probably small. The range is 1-22 for zstd and 1-9 for gzip.
NB2: The default program for gzip (at least with Debian) is the executable from zlib. With my workflows, libdeflate-gzip iscompatible and noticably faster.
NB3: This benchmark is 2 years old. The latest releases of zstd are much better, see https://github.com/facebook/zstd/releases
For a high compression, according to this benchmark xz can do slightly better, if you're willing to pay a 10× penalty on decompression.
xz -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]
zstd -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]
- Zstandard v1.5.6 – Chrome Edition
-
Optimizating Rabin-Karp Hashing
Compression, synchronization and backup systems often use rolling hash to implement "content-defined chunking", an effective form of deduplication.
In optimized implementations, Rabin-Karp is likely to be the bottleneck. See for instance https://github.com/facebook/zstd/pull/2483 which replaces a Rabin-Karp variant by a >2x faster Gear-Hashing.
- Show HN: macOS-cross-compiler – Compile binaries for macOS on Linux
-
Cyberpunk 2077 dev release
Get the data https://publicdistst.blob.core.windows.net/data/root.tar.zst magnet:?xt=urn:btih:84931cd80409ba6331f2fcfbe64ba64d4381aec5&dn=root.tar.zst How to extract https://github.com/facebook/zstd Linux (debian): `sudo apt install zstd` ``` tar -I 'zstd -d -T0' -xvf root.tar.zst ```
-
Honey, I shrunk the NPM package · Jamie Magee
I've done that experiment with zstd before.
https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...
Not sure about brotli though.
-
How in the world should we unpack archive.org zst files on Windows?
If you want this functionality in zstd itself, check this out: https://github.com/facebook/zstd/pull/2349
- Release Zstandard v1.5.5 · facebook/zstd
- ZSTD 1.5.5 is released with a corruption fix found at Google
-
zstd is used at Google
The story says : "ZSTD 1.5.5 is released with a corruption fix found at Google"
rsync
-
Which synchronization tool are you using together with the pCloud Crypto Folder?
rsync can be used to synchronize a local disk to the pCloud drive p. Works similarly as the Sync option of the pCloud Drive app. May be useful if one prefers a bulk upload once a day over a continuous synchronization.
-
Advanced Backup and Disaster Recovery Strategies
Tools that can be used to handle this include Rsync, Duplicati, Cohesity
-
Use rsync to Sync a Node Project to Dropbox and Ignore the node modules Folder
To overcome this issue, you can use rsync, a powerful command-line tool for synchronizing files and directories, along with a bash script that excludes the node_modules folder and also filter out anything in a .gitignore file that you specify. In this article, I'll guide you through the process of setting up and using this bash script to sync your Node.js project while ignoring the node_modules folder.
-
rsync error between windows and oracle linux
On OL9 I installed rsync using https://github.com/WayneD/rsync/blob/master/INSTALL.md and installing dependencies using the For Fedora 33 commands where possible except xxhash which wasn't found, so I used https://oraclelinux.pkgs.org/9/ol9-codeready-builder-aarch64/xxhash-devel-0.8.1-3.el9.aarch64.rpm.html
-
i have seen a video about syncthing and I am interested, but I would have a slightly different use case
rsync will probably work better for this use case
- How can I publish my vault to a website with community plugins?
-
Is there backup software that accounts for moved and renamed files?
I would prefer to use a revision control system, like git if keeping multiple versions of documents is important. Otherwise, if you don't find anything else to your liking, you could write your own scripts based on rsync with the clone-dest, detect-renamed, and detect-renamed-lax patches.
-
[Q] paperless-ngx: migrate to new host with UTF8 files?
Give rsync a try, it should handle UTF filenames just fine.
-
LXC Backup in mode "suspend" suddenly fails
While researching this issue I stumbled upon this thread where someone found out that a rsync 3.2.3 / glibc bug (github issue) leads to the error Operation not supported (95).\ my Proxmox instance is also using rsync 3.2.3: root@proxmox:~# rsync -V rsync version 3.2.3 protocol version 31 In this thread I found it's recommended to upgrade to another rsync version, but I'm not really sure how.\ \ Any thoughts on this?
- Google drive sync?
What are some alternatives?
LZ4 - Extremely Fast Compression algorithm
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
Snappy - A fast compressor/decompressor
syncthing-android - Wrapper of syncthing for Android.
LZMA - (Unofficial) Git mirror of LZMA SDK releases
libcurl - A command line tool and library for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, TFTP, WS and WSS. libcurl offers a myriad of powerful features
7-Zip-zstd - 7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard
dupeguru - Find duplicate files
ZLib - A massively spiffy yet delicately unobtrusive compression library.
rmlint - Extremely fast tool to remove duplicates and other lint from your filesystem
brotli - Brotli compression format
tailscale - The easiest, most secure way to use WireGuard and 2FA.