zfs-autosnap VS httm

I maintain two projects that I use daily for both work and personal stuff, that have attracted a modest, but appreciable amount of contributions. In both cases, the codebases are relatively small (500-1k sloc), and laser-focused on doing exactly one thing well.

I'm very grateful for every contribution, no matter how small - people have found bugs, fixed real problems, done cleanups. The hardest part is telling someone that a feature/idea does not have a place in this project. I think the general emphasis on minimalism tends to help here - I've never had to deal with any drama.

In terms of workload, again - the minimalist design and extremely clear goals have helped so much. I got trapped by that once before - I volunteered to build an internal automation tool (that saved someone else from doing like 1h/d of work), but literally couldn't spare 1h/mo to maintain it; the cause of the maintenance burden was an influx of changes in the APIs of the external services it integrated. So now I'm much more careful about volunteering to maintain integrations with external tools; in case of these two projects, the targets are SSH and ZFS - both have extremely stable interfaces.

In both cases it was absolutely worth it to publish and (very lightly) promote the projects; since these are "devops" tools that theoretically have unlimited potential for causing great harm, having any response at all helped reassure me that the code I'm running against production infrastructure has fewer unknown bugs. https://i.pinimg.com/474x/2f/e0/87/2fe08785e8eb112cada6da789...

The projects: <https://github.com/rollcat/judo>; <https://github.com/rollcat/zfs-autosnap>.

"This simple tool solves X at my org" is probably the most underrated type of project. There's not enough room to overcomplicate something that isn't a core part of the business, it must be practical to maintain, simple&stupid enough so that onboarding is not a hurdle, etc.

I encourage everyone to share your "splunk in 1kloc of Python" projects! Some of my own:

- https://github.com/rollcat/judo is Ansible without Python or YAML

- https://github.com/rollcat/zfs-autosnap manages rolling ZFS snapshots

A lot of these suggestions are heavily opinionated. Which is not necessarily bad, but they seem to mess with existing conventions just for the sake of it (why {pool}:{dataset}?).

> Don't make me name [...] snapshots.

You might like this little tool I wrote: https://github.com/rollcat/zfs-autosnap

You put "zfs-autosnap snap" in cron hourly (or however often you want a snapshot), and "zfs-autosnap gc" in cron daily, and it takes care of maintaining a rolling history of snapshots, per the retention policy.

It's not hard writing simple ZFS command wrappers, feel free to take my code and make your own tools.

My projects and many projects include a THIRD-PARTY-LICENSES.html file when I distribute binaries. See: https://github.com/kimono-koans/httm/blob/master/third_party/LICENSES_THIRD_PARTY.html

The only real advantage I can think of with nested ZFS is that the files in the KVM would obviously be individual inodes in the nested ZFS filesystem and datasets, in addition to the one inode per virtual volume on the hypervisor (or zvol). This would allow for granular file management on the kvm and the use of tools like https://github.com/kimono-koans/httm which is like a command line time machine for ZFS.

> It's worth noting that copy_file_range is used by a lot of things.

Yes, but the trigger feature, block cloning, only landed in the latest 2.2 release. If you immediately hopped on 2.2, and used a system with lots copy_file_range and FICLONE use, yes, you may have a problem (like, as you note, on Gentoo, where this problem surfaced).

Most people were just hopping on the bandwagon. My distro ships 2.1.5, so I have a 6 month wait until this feature lands, so I was just building copy_file_range support into my ZFS apps, right before news of this bug hit.[0]

> There are other things required to trigger the bug that are a lot less common though.

Exactly. My guess is the incidence of this will exceedingly rare for the common user/small NAS user/etc. I've run a corruption detector[0], and what I've found mostly indicates false positives. Some are build artifact fingerprints, which I don't care about, and which were deleted with the next build. The ones with an extant file on another system, I confirmed were a diff match with the origin using `rsync -rincv` and whats on snapshots with `httm --map-aliases`. So far no positive matches.

[0]: https://github.com/kimono-koans/httm

cargo install --git https://github.com/kimono-koans/httm --branch clones strace -f -o stderr.txt -e ioctl -- httm -r -R ~/.zshenv

httm prints the size, date and corresponding locations of available unique versions (deduplicated by modify time and size) of files residing on snapshots, but can also be used interactively to select and restore files, even snapshot mounts by file! httm might change the way you use snapshots (because ZFS/BTRFS/NILFS2 aren't designed for finding for unique file versions) or the Time Machine concept (because httm is very fast!).

All btrfs snapshot tools can have different layouts. It's mostly a nightmare for any one tool to support. Although its not the tool you're looking for, FYI AFAIK httm supports all/most btrfs layouts, but it took more work than necessary to get there.

ZFS only option which requires super user privileges.

[0]: https://github.com/kimono-koans/httm/blob/master/httm.1

httm - Interactive, file-level Time Machine-like tool for ZFS/btrfs/nilfs2