casync
duplicity
Our great sponsors
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
casync
-
Tool to clone file structure without the large files themselves?
You probably want casync.
-
LibSQL – a fork of SQLite that is both Open Source, and Open Contributions
(personally, I think more people need to be aware of casync for the update storage/distribution problem. It isn't perfect for every use case, but it's good enough that you're probably better off wrapping/forking it rather than reimplementing it badly from scratch)
-
improving download infra
Does something like casync (https://github.com/systemd/casync or https://github.com/folbricht/desync) serve any purpose or provide any advantage to propagating rpm changes over rsync?
-
Are there any true alternatives to Seafile? (Nextcloud is not an alternative in this context)
Software that comes to mind for syncing lots of small files: git (and other source versioning tools), casync (https://github.com/systemd/casync) and a go implementation (https://github.com/folbricht/desync). Not really an answer and I can't think of a way to shoehorn that into your workflow, but maybe it leads you down a useful road.
-
Hacker News top posts: Apr 23, 2022
Casync – A Content-Addressable Data Synchronization Tool\ (15 comments)
-
Casync – A Content-Addressable Data Synchronization Tool
I was wondering how this gets any common chunks at all with the removed file boundaries. Turns out that chunks don't have a set size, just min/max/avg values, so unaligned streams may end up synchronizing. https://github.com/systemd/casync/blob/master/src/cachunker.... If I understood that correctly, that's pretty cool.
But looking at the code I'm having strong "nope" feelings. First, because of lines like "q += m, n -= m;". Second, because of int/enum/semantic abuse: `compression_type` may be _CA_COMPRESSION_TYPE_INVALID which I hope is 0, `>= 0` as a known compression type, or `-EAGAIN` as an error. (from https://github.com/systemd/casync/blob/99559cd1d8cea69b30022... ) I'd bet that just throwing afl at the decompressor will find issues :(
I do like the idea though.
-
Blobcache is a content addressed data store, designed to be a replicated data layer for applications.
Compare https://github.com/systemd/casync which handles splitting/diffing, but does not handle fancy replication.
- Deduplicating Archiver with Compression and Encryption
duplicity
- Restic: Backups Done Right
- Deduplicating Archiver with Compression and Encryption
-
I recently learned about CHANGELOG and have a few questions about them
Well, depends on you. There are projects, that more or less put every commit into the CHANGELOG (there are even tools for this) or every PR. (Example)
-
Encrypted Backup Shootout
duplicity (python) - https://github.com/henrysher/duplicity
What are some alternatives?
BorgBackup - Deduplicating archiver with compression and authenticated encryption.
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
restic - Fast, secure, efficient backup program
kopia - Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Duplicati - Store securely encrypted backups in the cloud!
Bup - Very efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images). Please post problems or patches to the mailing list for discussion (see the end of the README below).
Duplicacy - A new generation cloud backup tool
tarsnap - Command-line client code for Tarsnap.
zstd - Zstandard - Fast real-time compression algorithm
desync - Alternative casync implementation