note-keeper VS dedupe

note-keeper shell script within a folder that is synced via nextcloud

Notekeeper - I wrote a simple tool for taking notes really quickly on the command line.

Notekeeper is a ~230 line long bash script that uses tools you already have installed to easily and quickly write and manage notes. Use your favorite editor and just start writing. Check out the github repo for more details and a complete list of features.

I wrote https://github.com/Gumnos/dedupe which sounds like it might be useful to you. It's faster than several of the alternatives I've found (many run the checksum across the whole of every file, this uses the file-size as a first-line discriminator, and only if the files are the same size does it go to the trouble of checking the checksum of the files). I designed it for creating hard-links in my media collection, but in the --dry-run mode, it should emit the file-names allowing you to pass it to xargs to remove them if it looks copacetic.

You can use my dedupe.py script with the dry-run flag (-n) to find all the duplicates on your drive. If you run it without the dry-run flag, it will attempt to make hard-links so that each file exists only once on the drive with multiple hard-links to the underlying file. It should be pretty fast, only needing to checksum file-content in the event that files have the same size (several other such deduplication methods work by checksumming every file on the drive which can be slow).

a file-deduplication utility that hard-links duplicate files to save space (our family photo gallery gets pics put in multiple albums for various audiences, so I can cut down on a lot of duplication with this)