zfsbootmenu VS httm

We use a couple nice home-grown functions in ZFSBootMenu to help debug things. We have a zdebug logging function that's peppered liberally throughout the code base - https://github.com/zbm-dev/zfsbootmenu/blob/master/zfsbootme...

Hitting ctrl-t on our main menu will, when booting with debug logging enabled, show a screen like this: https://imgur.com/Ge75zkP

We also have a flamegraph profiling mechanism that can be enabled with https://github.com/zbm-dev/zfsbootmenu/blob/master/zfsbootme... . That will dump data to a serial port, which when re-assembled, can be used to produce a graph like https://raw.githubusercontent.com/zbm-dev/zfsbootmenu/master...

Bash is suprisingly flexible.

A lot of what's in the Pure Bash Bible is horrifically slow. Many of those things are substantially faster, even when paying the cost of starting a new process, when you use an external and commonly available tool. I wrote a bash performance profiler that outputs data in a format that flamegraph.pl recognizes - it really helped identify where we could improve the performance of ZFSBootMenu.

https://github.com/zbm-dev/zfsbootmenu/releases/tag/v1.12.0

Don't fall in the trap of thinking things have to be written entirely in bash; it's okay to use other tools to help fill in the gaps.

If instead of "opensuse" you're asking for bootloader as grub can't boot from zfs, then, like i metnioned, i don't use grub2, i uninstalled it, instead i'm using https://github.com/zbm-dev/zfsbootmenu

I thought the following was supposed to fix this issue: https://github.com/zbm-dev/zfsbootmenu/commit/84da18e64ebcc0c483e7b2c7d3972f7d91784e63

All release assets, including EFI executables and kernel/initramfs pairs, are signed with signify, which provides a simple method for verifying that the contents of the file are as this project intended. Once you've installed signify (that's left as an exercise, although Void Linux provides the signify package for this purpose), just download the desired assets from the ZFSBootMenu release page, download the file sha256.sig alongside it, and run:

I think the consensus is that you probably shouldn't be installing a ZFS on root using the native installer anymore. They aren't really maintaining the packages that make that work. Instead the suggestion is to go the zfsbootmenu route of installing.

Glad to hear that everything is working for you! I've opened a PR that adds a warning about this condition - it should likely make it into 2.2.0.

You can install following instructions at https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Bullseye%20Root%20on%20ZFS.html which I've automated with https://github.com/HankB/Linux_ZFS_Root/tree/master/Debian. For scripting, you should also look at https://github.com/zbm-dev/zfsbootmenu. I'd probably go that way if I were starting from scratch.

ZBM provides an amazingly useful script in it's wiki here. This runs when a new kernel is updated by xbps and it snapshots your system before the kernel is installed. This creates a boot environment, and via the magic of ZFS boot environments, allows you to rollback any kernel update to a known, working configuration.

* https://docs.oracle.com/cd/E86824_01/html/E54764/beadm-1m.ht...

> A ZFS boot environment is a bootable clone of the datasets needed to boot the operating system. Creating a BE before performing an upgrade provides a low-cost safeguard: if there is a problem with the update, the system can be rebooted back to the point in time before the upgrade.

* https://klarasystems.com/articles/managing-boot-environments...

Or perhaps:

> In essence, ZFSBootMenu is a small, self-contained Linux system that knows how to find other Linux kernels and initramfs images within ZFS filesystems. When a suitable kernel and initramfs are identified (either through an automatic process or direct user selection), ZFSBootMenu launches that kernel using the kexec command.

* https://github.com/zbm-dev/zfsbootmenu

My projects and many projects include a THIRD-PARTY-LICENSES.html file when I distribute binaries. See: https://github.com/kimono-koans/httm/blob/master/third_party/LICENSES_THIRD_PARTY.html

The only real advantage I can think of with nested ZFS is that the files in the KVM would obviously be individual inodes in the nested ZFS filesystem and datasets, in addition to the one inode per virtual volume on the hypervisor (or zvol). This would allow for granular file management on the kvm and the use of tools like https://github.com/kimono-koans/httm which is like a command line time machine for ZFS.

> It's worth noting that copy_file_range is used by a lot of things.

Yes, but the trigger feature, block cloning, only landed in the latest 2.2 release. If you immediately hopped on 2.2, and used a system with lots copy_file_range and FICLONE use, yes, you may have a problem (like, as you note, on Gentoo, where this problem surfaced).

Most people were just hopping on the bandwagon. My distro ships 2.1.5, so I have a 6 month wait until this feature lands, so I was just building copy_file_range support into my ZFS apps, right before news of this bug hit.[0]

> There are other things required to trigger the bug that are a lot less common though.

Exactly. My guess is the incidence of this will exceedingly rare for the common user/small NAS user/etc. I've run a corruption detector[0], and what I've found mostly indicates false positives. Some are build artifact fingerprints, which I don't care about, and which were deleted with the next build. The ones with an extant file on another system, I confirmed were a diff match with the origin using `rsync -rincv` and whats on snapshots with `httm --map-aliases`. So far no positive matches.

[0]: https://github.com/kimono-koans/httm

cargo install --git https://github.com/kimono-koans/httm --branch clones strace -f -o stderr.txt -e ioctl -- httm -r -R ~/.zshenv

httm prints the size, date and corresponding locations of available unique versions (deduplicated by modify time and size) of files residing on snapshots, but can also be used interactively to select and restore files, even snapshot mounts by file! httm might change the way you use snapshots (because ZFS/BTRFS/NILFS2 aren't designed for finding for unique file versions) or the Time Machine concept (because httm is very fast!).

All btrfs snapshot tools can have different layouts. It's mostly a nightmare for any one tool to support. Although its not the tool you're looking for, FYI AFAIK httm supports all/most btrfs layouts, but it took more work than necessary to get there.

ZFS only option which requires super user privileges.

[0]: https://github.com/kimono-koans/httm/blob/master/httm.1

httm - Interactive, file-level Time Machine-like tool for ZFS/btrfs/nilfs2