ccheck
ccheck | zfs-to-aws | |
---|---|---|
5 | 4 | |
26 | 28 | |
- | - | |
0.0 | 1.8 | |
about 3 years ago | over 2 years ago | |
Perl | Shell | |
MIT License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
ccheck
-
Anyone know what causes intermittent corruption of random visual media files across drives and machines?
Grab a friends computer and amass a large batch of good known files, make sure they are of all different file formats. I am pretty sure you will be able to find entire archives of test data in different formats online, to really reproduce this I am going to assume it should be multiple gb in size. Make sure it contains jpg, videos, text files, pdfs, etc. Now write a script or use some tool like this (https://github.com/jwr/ccheck) to basically compute the sha256 checksum of every file in this test package and write it to a file. Take this package of files and copy them to as many media sources as you have access to, CD/DVDs are great, thumb drive, your laptop, a nas with ZFS (and ECC ram) would be amazing, probably throw it up on cloud storage just to be safe. I would then have the same script run as a cron job, maybe on your main machine to basically continuously check that checksums match their original value. As soon as you notice a checksum mismatch you will want to isolate that file and locate the same one across all the other systems and do a deeper inspection. Open it up in a HEX editor and do a bit by bit comparison to see were the corruption occurred and how bad it is. This will start to give you a better picture of what may be going on.
-
Show HN: Off-site, encrypted backups for $1/TB/month at 99.999999999% durability
Here's my "me too" — I've been happily using rclone for things like photo archives (together with my small consistency checker to check file hashes for corruption https://github.com/jwr/ccheck). I also use Arq Backup with B2 as the destination. This gives me very reasonable storage costs and backups I can access and test regularly.
-
What Happened to Perl 7?
Perl is very well suited for certain tasks (not large software systems, but programs that process data). It is also one of very few languages/ecosystems where you can expect your code to work after >10 years. This is why I sometimes use it, for example my fs consistency checker (https://github.com/jwr/ccheck) was written in Perl specifically because it's a long-term tool and I would like to be able to run it on any system in 15 years.
Compare this long-term approach with the fires in Python or (heaven forbid) Node ecosystems, where things break all the time.
-
I Nearly Lost the Lightroom Catalog with All My Photos
This sort of thing scares me. It's why I started running consistency checks on my important archives (like my photo library), which I keep backed up in multiple places. We tend to think that in a digital world bits are just bits and do not get corrupted — which is decidedly untrue.
I wrote my own consistency checker, as I wasn't happy with what was out there. I wanted it to be simple, and maintainable in the long term (>10 years horizon). See https://github.com/jwr/ccheck if you need something like this. I now update my checksums regularly and check for corruption.
-
How do I safely store my files?
Good point about bitrot. This is why I wrote ccheck.pl (https://github.com/jwr/ccheck) — I wanted to be able to check and detect bitrot in a way that depends on as little technology as possible.
zfs-to-aws
-
Show HN: Off-site, encrypted backups for $1/TB/month at 99.999999999% durability
I've created similar functionality just using bash that will send the lates version of ZFS datasets to S3/Glacier, including dealing with incremental changes. I have mentioned this previously on HN and got a few useful changes submitted for it, especially making it more platform agnostic.
I have some open tickets asking about restoring. I haven't tried this yet as this has been a backup of last resort for me, but hopefully posting this again will nudge me into looking at that.
https://github.com/agurk/zfs-to-aws/
- ZFS cloud storage
- Agurk/ZFS-to-AWS
-
Welcome Z3 – ZFS to S3 Backup/Restore Tool
For anyone who might be interested in doing this with fewer dependencies/tooling I have written a bash script that pushes ZFS snapshots to S3:
https://github.com/agurk/zfs-to-aws
I have it running in a cronjob on my NAS nightly, and a retention policy on aws to push it straight into glacier.
There's no restore script as this is my last line of backups, so I thought if I was restoring from here I'd want to do it manually. I might get around to writing one now however.
What are some alternatives?
glacier_deep_archive_backup - Extremely low cost, off-site backup/restore using AWS S3 Glacier Deep Archive
voidvault - Bootstrap Void with FDE
z3 - Backup your ZFS snapshots to S3.
darktable - darktable is an open source photography workflow application and raw developer
zfs - OpenZFS on Linux and FreeBSD
App-perlbrew - Manage perl installations in your $HOME
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
berrybrew - Perlbrew for Windows!
zfs-to-glacier - A tool to sync zfs snapshots to s3-glacier, written in rust.
roast - 🦋 Raku test suite
zfs3backup - Backup zfs snapshots to S3.