rmlint VS zfs

My preferred solution is rmlint [https://github.com/sahib/rmlint] mostly because it also looks at duplicate directories. It produces a bash script instead of deleting anything itself, so you can examine it before running the script it made.

After I removed duplicates (with help of https://github.com/sahib/rmlint ), I migrated my photos to an ordinary zpool instead.

For the technically savvy among you there is an excellent open source program called ‘rmlint’ (aka. Remove Lint). It is excellent at finding duplicates and saved me terabytes of space.

You don’t say if you are on Windows or Unix. I have used rmlint successfully in the past.

dupeguru or rmlint

rmlint, my friend, is the last tool you will ever need for this

I use and test assorted duplicate finders regularly.

fdupes is the classic (going way way back) but it's really very slow, not worth using anymore.

The four I know are worth trying these days (depending on data set, hardware, file arrangement and other factors, any one of these might be fastest for a specific use case) are https://github.com/jbruchon/jdupes , https://github.com/pauldreik/rdfind , https://github.com/jvirkki/dupd , https://github.com/sahib/rmlint

Had not encountered fclones before, will give it a try.

Be careful if you use ZFS-on-root, make sure not to snapshot bpool or it will brick your system and require a complete reinstall.

https://github.com/openzfs/zfs/issues/13873

> The only non-junk PCIe3 option that's even advertised here recently is the overpriced WD Red SN700.

Those WD drives seem to have some real issues, at least with ZFS and btrfs. :(

https://github.com/openzfs/zfs/discussions/14793

IMO the "next fs" is just zfs. They somewhat recently merged RAIDZ expansion feature https://github.com/openzfs/zfs/pull/12225 and make regular improvements. If no file system has what you need today, zfs will probably be the first one to have it "tomorrow," imo.

https://github.com/openzfs/zfs/issues/15526#issuecomment-181...

> zpool get all tank | grep bclone

> kc3000 bcloneused 442M

> kc3000 bclonesaved 1.42G

> kc3000 bcloneratio 4.30x

> My understanding is this: If the result is 0 for both bcloneused and bclonesaved then it's safe to say that you don't have silent corruption.

A couple years ago, I had an idea for convincing a filesystem to go faster using 2 compression steps instead of one. I couldn't see why it wouldn't work, and I also couldn't convince myself it should.

It seems to have worked out. [1]

[1] - https://github.com/openzfs/zfs/commit/f375b23c026aec00cc9527...

https://github.com/openzfs/zfs/issues/7631

This is a long-standing issue with zvols which affects overall system stability, and has no real solution as of yet.