casync
kopia
Our great sponsors
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
casync
-
Tool to clone file structure without the large files themselves?
You probably want casync.
-
LibSQL – a fork of SQLite that is both Open Source, and Open Contributions
(personally, I think more people need to be aware of casync for the update storage/distribution problem. It isn't perfect for every use case, but it's good enough that you're probably better off wrapping/forking it rather than reimplementing it badly from scratch)
-
improving download infra
Does something like casync (https://github.com/systemd/casync or https://github.com/folbricht/desync) serve any purpose or provide any advantage to propagating rpm changes over rsync?
-
Are there any true alternatives to Seafile? (Nextcloud is not an alternative in this context)
Software that comes to mind for syncing lots of small files: git (and other source versioning tools), casync (https://github.com/systemd/casync) and a go implementation (https://github.com/folbricht/desync). Not really an answer and I can't think of a way to shoehorn that into your workflow, but maybe it leads you down a useful road.
-
Hacker News top posts: Apr 23, 2022
Casync – A Content-Addressable Data Synchronization Tool\ (15 comments)
-
Casync – A Content-Addressable Data Synchronization Tool
I was wondering how this gets any common chunks at all with the removed file boundaries. Turns out that chunks don't have a set size, just min/max/avg values, so unaligned streams may end up synchronizing. https://github.com/systemd/casync/blob/master/src/cachunker.... If I understood that correctly, that's pretty cool.
But looking at the code I'm having strong "nope" feelings. First, because of lines like "q += m, n -= m;". Second, because of int/enum/semantic abuse: `compression_type` may be _CA_COMPRESSION_TYPE_INVALID which I hope is 0, `>= 0` as a known compression type, or `-EAGAIN` as an error. (from https://github.com/systemd/casync/blob/99559cd1d8cea69b30022... ) I'd bet that just throwing afl at the decompressor will find issues :(
I do like the idea though.
-
Blobcache is a content addressed data store, designed to be a replicated data layer for applications.
Compare https://github.com/systemd/casync which handles splitting/diffing, but does not handle fancy replication.
- Deduplicating Archiver with Compression and Encryption
kopia
-
I Backup
I've been happy with: https://kopia.io/
Fairly easy to configure, does snapshots to S3 and has a icon in my tray I can watch :)
-
Kopia: Open-Source, Fast and Secure Open-Source Backup Software
I noticed this project while comparing restic/borg and am thinking about trying it.
Initially I thought this was a corporate project and was looking for the monetization model, but then I found https://github.com/kopia/kopia/blob/master/GOVERNANCE.md
I feel like the project might benefit from making their governance model more prominent on the website.
Kopia also means a copy in Polish and the author is Polish. The first paragraph in the software's Github page also confirms the Polish origin of the name: https://github.com/kopia/kopia/
Tangentially, as far as OSS names of Polish go, kopia is pretty tame. A popular UI deduplicating app is called czkawka (hiccup). Now that choice is just mean towards non-Polish speakers. :)
Kopia is great, though it's worth noting for folks on Linux: non-UTF-8 paths aren't stored correctly [1] and xattrs aren't stored [2]. While most folks probably won't care about the former, the latter can could cause issues (eg. losing SELinux labels makes it difficult to restore a backup of the root filesystem on distros that use SELinux).
-
Borgbase backups have been unavailable for 3 days – recovery is at 26%
I used their trial for a bit to test it out with Vorta [1] in a container. Vorta (and Borg) seemed to work fine, until I wanted to restore an archive and I noticed that my recent snapshots were completely empty. Probably because of a misconfiguration on my end though. But it made me look elsewhere. For me backups should be a fire, test and forget solution.
Recently I made the switch to Kopia [2] which seems to have feature parity with Borg (and Restic [3]). It also has a web UI which is way easier to work with than Vorta. And I can easily view, extract and restore individual files or folders from there. This gave me way more confidence about this solution. The only thing I really miss is that I cannot chose different targets for different paths. For instance, with Borg I was able to backup a partial of my Docker appdata to an external source. And I haven't found a way to do this with Kopia. Besides that I'm pretty happy with this solution and I would recommend it.
-
Show HN: Gdańsk AI – full stack AI voice chatbot (STT, LLM, TTS, auth, payments)
There's a few. Off the top of my head
-
Is there a good way to "roll back" a failed attempt at upgrading to Debian 12?
Backups, Backups, Backups.
-
Not openSUSE specific but what's the best backup utility?
Kopia, it has an AppImage version that works on openSUSE.
-
About Scripts for Btrfs Maintenance
Surely this incident highlights the importance of backups, right? 5 TB is even a manageable amount of data.
I also used to run btrfs in btrfs-RAID10 configuration until apparently a flapping SATA link and fsck attempts were able to break the fs completely. Full system backups were great that day. I run https://kopia.io/ nowadays every three hours during day time and I've been quite happy with it.
Nowadays I run bcachefs.. Backups are still handy :).
I suppose the reason why you chose NTFS was to be able to access the data from Windows, at least in case of emergency? Because there are a lot of filesystems that are presumably more mature than NTFS is for Linux.
What are some alternatives?
restic - Fast, secure, efficient backup program
BorgBackup - Deduplicating archiver with compression and authenticated encryption.
Duplicati - Store securely encrypted backups in the cloud!
UrBackup - UrBackup - Client/Server Open Source Network Backup for Windows, MacOS and Linux
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
restic-wrapper - Simple bash wrapper to source .env configuration files for Restic. Facilitates both manual CLI execution and scheduled (cron) execution.
vorta - Desktop Backup Client for Borg Backup
Duplicacy - A new generation cloud backup tool
docker-volume-backup - Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage, Dropbox or SSH compatible storage
bupstash - Easy and efficient encrypted backups.
Blobbackup - Private, Secure Computer Backups
corso - Free, Secure, and Open-Source Backup for Microsoft 365