czkawka VS AntiDupl

You want content-addressed storage; this works with rolling content hashes that identify common blocks of memory. `rsync` uses that technique to minimize bytes to be transferred. https://github.com/qarmin/czkawka is a GUI app and CLI tool to find identical files in general and similar images in particular.

The task is much simpler if you only want to find bit-identical entire files, not part of files; in that case, you can just run a tool like `sha1sum` over each file and record the hash digest in a database; identical files—and only identical ones, with high probability—will have the same hash, non-identical ones will have different hashes.

You might be interested in this app: https://github.com/qarmin/czkawka

https://github.com/qarmin/czkawka by far the best of anything iv tried

I've used Czkawka (https://github.com/qarmin/czkawka) because it does Lanczos-based image duplicate detection, which makes it more practical for me.

Maybe it's a setting you made or the files, not sure. You can try another software czkawka to see if you get better results with it.

For static images i used https://github.com/qarmin/czkawka and it works well enough. I think. But when i used it on a folder with gifs and their jxl conversions, it shows nothing. SURELY this could not be user error, rrrright?

I used to use DupeGuru which has some photo-specific dupe detection where you can fuzzy match image dupes based on content: https://dupeguru.voltaicideas.net/

But I switched over to czkawka, which has a better interface for comparing files, and seems to be a bit faster: https://github.com/qarmin/czkawka

Unfortunately, neither of these are integrated into Photoprism, so you still have to do some file management outside the database before importing.

I also haven't used Photoprism extensively yet (I think it's running on one of my boxes, but I haven't gotten around to setting it up), but I did find that it wasn't really built for file-based libraries. It's a little more heavyweight, but my research shows that Nextcloud Memories might be a better choice for me (it's not the first-party Nextcloud photos app, but another one put together by the community): https://apps.nextcloud.com/apps/memories

AntiDupl.NET can list duplicates for you, and you can use easy keybindings to quickly decide which duplicate to keep and which to delete. Along with quality estimates, say, if you had two identical pictures but one was high-quality and one low-quality jpg.

I'm looking for software that could look at a collection of MP3 files, and list the ones having very similar audio content. Exactly like AntiDupl does for images.

If you mean locally, AntiDupl has been fantastic for me.

Download it from the source: https://github.com/ermig1979/AntiDupl

Yes, czkawka is pretty good. If /u/AndypandyO is also interested in de-duplicating similar (but not exact) pictures, I recommend AntiDupl.

Antidupl won't move them to a folder, but it will display them to you, so that you can compare each pair of duplicates by size/resolution, blockiness etc., so that you can delete the one(s) you don't need. Exact matches can be deleted automatically, if you select this option.

AntiDupl.NET program is free and open-source software

AntiDupl