Tummy-backup Alternatives

tummy-backup reviews and mentions

• A almost perfect rsync over SSH backup script
  7 projects | news.ycombinator.com | 25 Feb 2022

  I've been doing rsync-based backups of close to a thousand systems for ~20 years, most notably for a long time I backed up the python.org infrastructure, and I have quite a few thoughts on this. I also have a battle-tested rsync wrapper that I'll point to below.

  - Backups should be automatic, only requiring attention when it is needed. This script philosophy seems to be "Just do your best, mail a log file, and rely on the user to figure out if something didn't work". Even for home backups, this is just wrong.

  - As an example of the above: This script notes that it fails if a backup takes more than 24 hours.

  - The "look for other rsyncs running" part of the code is an odd way of approaching locking, but for a single personal "push" backup I guess it is ok.

  I've got an rsync wrapper that has been battle tested over a couple decades and hundreds of servers here: https://github.com/linsomniac/tummy-backup/blob/master/sbin/...

  Features of it are:

  - As the filename implies, the goal is to rsync to a zfs destination, and it will take a zfs snapshot as part of this. It is easy to customize to another backup destination, I've had people report they have customized it for their own laptop backups, for example to an rsync.net destination.

  - It goes out of its way to detect when rsync has failed and log that.

  - It does do "inplace" rsyncs, which dramatically save space if you have large files that get appended to (logfiles, ZODB databases).

  - This is part of a larger system that manages the rsyncs of multiple systems, both local and remote. Things like alerting are done if a backup of a system has failed consistently for many days.

  - In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails.

  The hardlink trick only works for fairly small data sets. Issues include: Managing hard links takes a lot of overhead, especially on spinning discs. Large files being appended to use a ton of space (a 4GB file with 1K appended every day uses 128GB to store 14 dailies, 6 weeklies, and 12 monthlies). ZFS is a pretty good destination for rsync, as similar snapshots will use 4GB to store.