czkawka VS datacurator-filetree

You want content-addressed storage; this works with rolling content hashes that identify common blocks of memory. `rsync` uses that technique to minimize bytes to be transferred. https://github.com/qarmin/czkawka is a GUI app and CLI tool to find identical files in general and similar images in particular.

The task is much simpler if you only want to find bit-identical entire files, not part of files; in that case, you can just run a tool like `sha1sum` over each file and record the hash digest in a database; identical files—and only identical ones, with high probability—will have the same hash, non-identical ones will have different hashes.

You might be interested in this app: https://github.com/qarmin/czkawka

https://github.com/qarmin/czkawka by far the best of anything iv tried

I've used Czkawka (https://github.com/qarmin/czkawka) because it does Lanczos-based image duplicate detection, which makes it more practical for me.

Maybe it's a setting you made or the files, not sure. You can try another software czkawka to see if you get better results with it.

For static images i used https://github.com/qarmin/czkawka and it works well enough. I think. But when i used it on a folder with gifs and their jxl conversions, it shows nothing. SURELY this could not be user error, rrrright?

I used to use DupeGuru which has some photo-specific dupe detection where you can fuzzy match image dupes based on content: https://dupeguru.voltaicideas.net/

But I switched over to czkawka, which has a better interface for comparing files, and seems to be a bit faster: https://github.com/qarmin/czkawka

Unfortunately, neither of these are integrated into Photoprism, so you still have to do some file management outside the database before importing.

I also haven't used Photoprism extensively yet (I think it's running on one of my boxes, but I haven't gotten around to setting it up), but I did find that it wasn't really built for file-based libraries. It's a little more heavyweight, but my research shows that Nextcloud Memories might be a better choice for me (it's not the first-party Nextcloud photos app, but another one put together by the community): https://apps.nextcloud.com/apps/memories

For the most part I follow this file tree. However when it comes to some of my intererests, like electronics, I am unsure if I should keep splitting these interest-based files by file type, for example:

I have redesigned my entire computer to follow the datacurator methodology: https://github.com/roboyoshi/datacurator-filetree/tree/main/root

P.S. I've been lurking this sub and have considered this particular problem for a long time and have read maybe everything Karl, Nayuki, Reddit, and Hacker News have had to say on the subject. Running into this post is a treat. If tags don't work out for you roboyoshi and contributors have made a really nice unified file tree https://github.com/roboyoshi/datacurator-filetree

As always, credit to u/Roboyoshi for the Datacurator filetree.

But it will obviously depend on the use case. As example you have JohnnyDecimal or a more simple approach

This tool will be useful to generate repos like these and sharing them with friends without actually needing to share them TB of data.

back in the days I started with this structure tho: https://github.com/roboyoshi/datacurator-filetree

This is a mix of this post and https://github.com/roboyoshi/datacurator-filetree. Im still having trouble with a few things: