rmlint VS zfs

My preferred solution is rmlint [https://github.com/sahib/rmlint] mostly because it also looks at duplicate directories. It produces a bash script instead of deleting anything itself, so you can examine it before running the script it made.

After I removed duplicates (with help of https://github.com/sahib/rmlint ), I migrated my photos to an ordinary zpool instead.

For the technically savvy among you there is an excellent open source program called ‘rmlint’ (aka. Remove Lint). It is excellent at finding duplicates and saved me terabytes of space.

You don’t say if you are on Windows or Unix. I have used rmlint successfully in the past.

dupeguru or rmlint

rmlint, my friend, is the last tool you will ever need for this

I use and test assorted duplicate finders regularly.

fdupes is the classic (going way way back) but it's really very slow, not worth using anymore.

The four I know are worth trying these days (depending on data set, hardware, file arrangement and other factors, any one of these might be fastest for a specific use case) are https://github.com/jbruchon/jdupes , https://github.com/pauldreik/rdfind , https://github.com/jvirkki/dupd , https://github.com/sahib/rmlint

Had not encountered fclones before, will give it a try.

Not in production, but using ZoL on my personal workstations. https://zfsonlinux.org/

Some discussion: https://www.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_...

You could clone this branch, build it (you don't need to install it) and then do something like ./zdb -L -Q foo/bar or zdb -L -Q foo/ if you want the whole pool. (Obviously, if you're walking all the metadata, even with a special vdev, that can take a bit.)

That seems reasonable to me. Maybe a different ordering would be better, though, and I need to make sure I'm not missing any subtleties...