ohmycron
tummy-backup
ohmycron | tummy-backup | |
---|---|---|
1 | 1 | |
138 | 14 | |
0.7% | - | |
0.0 | 0.0 | |
about 6 years ago | over 2 years ago | |
Shell | Python | |
ISC License | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
ohmycron
-
A almost perfect rsync over SSH backup script
> # avoidng collisions with other rsync processes
Use https://github.com/instacart/ohmycron
> MONTHROTATE=monthrotate # use DD instead of YYMMDD
Use https://rotate-backups.readthedocs.io/en/latest/readme.html
> $RSYNC -avR "$SOURCE" "${RSYNCCONF[@]}"
Just create a one-line script with the hardcoded rsync command you want to use and replace the directory to sync as a command-line argument, e.g.
#!/bin/sh
tummy-backup
-
A almost perfect rsync over SSH backup script
I've been doing rsync-based backups of close to a thousand systems for ~20 years, most notably for a long time I backed up the python.org infrastructure, and I have quite a few thoughts on this. I also have a battle-tested rsync wrapper that I'll point to below.
- Backups should be automatic, only requiring attention when it is needed. This script philosophy seems to be "Just do your best, mail a log file, and rely on the user to figure out if something didn't work". Even for home backups, this is just wrong.
- As an example of the above: This script notes that it fails if a backup takes more than 24 hours.
- The "look for other rsyncs running" part of the code is an odd way of approaching locking, but for a single personal "push" backup I guess it is ok.
I've got an rsync wrapper that has been battle tested over a couple decades and hundreds of servers here: https://github.com/linsomniac/tummy-backup/blob/master/sbin/...
Features of it are:
- As the filename implies, the goal is to rsync to a zfs destination, and it will take a zfs snapshot as part of this. It is easy to customize to another backup destination, I've had people report they have customized it for their own laptop backups, for example to an rsync.net destination.
- It goes out of its way to detect when rsync has failed and log that.
- It does do "inplace" rsyncs, which dramatically save space if you have large files that get appended to (logfiles, ZODB databases).
- This is part of a larger system that manages the rsyncs of multiple systems, both local and remote. Things like alerting are done if a backup of a system has failed consistently for many days.
- In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails.
The hardlink trick only works for fairly small data sets. Issues include: Managing hard links takes a lot of overhead, especially on spinning discs. Large files being appended to use a ton of space (a 4GB file with 1K appended every day uses 128GB to store 14 dailies, 6 weeklies, and 12 monthlies). ZFS is a pretty good destination for rsync, as similar snapshots will use 4GB to store.
What are some alternatives?
minicron - 🕰️ Monitor your cron jobs
kopia - Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
borg-cron-helper - Helper shell scripts for BorgBackup to automate backups and make your life easier… 😉
restic - Fast, secure, efficient backup program
QuartzNet - Quartz Enterprise Scheduler .NET
synbak - Synbak - Universal Backup System
Rsnapshot - a tool for backing up your data using rsync (if you want to get help, use https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss)