zfs-to-aws VS ccheck

I've created similar functionality just using bash that will send the lates version of ZFS datasets to S3/Glacier, including dealing with incremental changes. I have mentioned this previously on HN and got a few useful changes submitted for it, especially making it more platform agnostic.

I have some open tickets asking about restoring. I haven't tried this yet as this has been a backup of last resort for me, but hopefully posting this again will nudge me into looking at that.

https://github.com/agurk/zfs-to-aws/

For anyone who might be interested in doing this with fewer dependencies/tooling I have written a bash script that pushes ZFS snapshots to S3:

https://github.com/agurk/zfs-to-aws

I have it running in a cronjob on my NAS nightly, and a retention policy on aws to push it straight into glacier.

There's no restore script as this is my last line of backups, so I thought if I was restoring from here I'd want to do it manually. I might get around to writing one now however.

Grab a friends computer and amass a large batch of good known files, make sure they are of all different file formats. I am pretty sure you will be able to find entire archives of test data in different formats online, to really reproduce this I am going to assume it should be multiple gb in size. Make sure it contains jpg, videos, text files, pdfs, etc. Now write a script or use some tool like this (https://github.com/jwr/ccheck) to basically compute the sha256 checksum of every file in this test package and write it to a file. Take this package of files and copy them to as many media sources as you have access to, CD/DVDs are great, thumb drive, your laptop, a nas with ZFS (and ECC ram) would be amazing, probably throw it up on cloud storage just to be safe. I would then have the same script run as a cron job, maybe on your main machine to basically continuously check that checksums match their original value. As soon as you notice a checksum mismatch you will want to isolate that file and locate the same one across all the other systems and do a deeper inspection. Open it up in a HEX editor and do a bit by bit comparison to see were the corruption occurred and how bad it is. This will start to give you a better picture of what may be going on.

Here's my "me too" — I've been happily using rclone for things like photo archives (together with my small consistency checker to check file hashes for corruption https://github.com/jwr/ccheck). I also use Arq Backup with B2 as the destination. This gives me very reasonable storage costs and backups I can access and test regularly.

Perl is very well suited for certain tasks (not large software systems, but programs that process data). It is also one of very few languages/ecosystems where you can expect your code to work after >10 years. This is why I sometimes use it, for example my fs consistency checker (https://github.com/jwr/ccheck) was written in Perl specifically because it's a long-term tool and I would like to be able to run it on any system in 15 years.

Compare this long-term approach with the fires in Python or (heaven forbid) Node ecosystems, where things break all the time.

This sort of thing scares me. It's why I started running consistency checks on my important archives (like my photo library), which I keep backed up in multiple places. We tend to think that in a digital world bits are just bits and do not get corrupted — which is decidedly untrue.

I wrote my own consistency checker, as I wasn't happy with what was out there. I wanted it to be simple, and maintainable in the long term (>10 years horizon). See https://github.com/jwr/ccheck if you need something like this. I now update my checksums regularly and check for corruption.

Good point about bitrot. This is why I wrote ccheck.pl (https://github.com/jwr/ccheck) — I wanted to be able to check and detect bitrot in a way that depends on as little technology as possible.