dano VS czkawka

You should check out my program dano. It has a dupe detection function.

dano is a wrapper for ffmpeg that checksums the internal file streams of ffmpeg compatible media files, and stores them in a format which can be used to verify such checksums later. This is handy, because, should you choose to change metadata tags, or change file names, the media checksums should remain the same.

dano - a hashdeep/md5tree (but much more) for media files

My best advice (from a person who knows only enough to package his own projects) is it depends on the project. For my own Rust projects, I've found a Debian autopackager called cargo-deb for cargo. I convert those deb packages to rpm with alien. I upload both to the repo manually, and have a script which generates what I need and uploads to a PPA.

The meaty part of the holidays comes a little early this year for Ubuntu/Debian users, as I've prepared a special treat -- a PPA for httm (and dano)!

If you use something like ZFS, then you have a system thats watching your entire data path. If you're not paranoid, afraid of other ways your data can be corrupted, bit rot, etc. (again you do you), you might still want to look at a program I wrote awhile ago, dano, and have a system to periodically verify the data on disk.

One thing ZFS doesn't do (and thank goodness it doesn't?), it doesn't detect out-of-band changes to files. If a program has the permissions to modify a file, ZFS happily modifies that file. But, for some files (and for the truly paranoid), we may want to keep additional metadata about file integrity. For instance, FLAC files keep checksums of its music streams. A program I wrote, dano, provides a way to do the same for all FFMPEG compatible media streams.

Packages and source available for dano at the following link.

https://github.com/qarmin/czkawka by far the best of anything iv tried

I've used Czkawka (https://github.com/qarmin/czkawka) because it does Lanczos-based image duplicate detection, which makes it more practical for me.

Maybe it's a setting you made or the files, not sure. You can try another software czkawka to see if you get better results with it.

For static images i used https://github.com/qarmin/czkawka and it works well enough. I think. But when i used it on a folder with gifs and their jxl conversions, it shows nothing. SURELY this could not be user error, rrrright?

I used to use DupeGuru which has some photo-specific dupe detection where you can fuzzy match image dupes based on content: https://dupeguru.voltaicideas.net/

But I switched over to czkawka, which has a better interface for comparing files, and seems to be a bit faster: https://github.com/qarmin/czkawka

Unfortunately, neither of these are integrated into Photoprism, so you still have to do some file management outside the database before importing.

I also haven't used Photoprism extensively yet (I think it's running on one of my boxes, but I haven't gotten around to setting it up), but I did find that it wasn't really built for file-based libraries. It's a little more heavyweight, but my research shows that Nextcloud Memories might be a better choice for me (it's not the first-party Nextcloud photos app, but another one put together by the community): https://apps.nextcloud.com/apps/memories

Czkawka can do this.

There's always the well-respected tool, Czkawka. Or, of the CLI is your thing, jdupes is a good option.