dedupe VS command_help

I wrote https://github.com/Gumnos/dedupe which sounds like it might be useful to you. It's faster than several of the alternatives I've found (many run the checksum across the whole of every file, this uses the file-size as a first-line discriminator, and only if the files are the same size does it go to the trouble of checking the checksum of the files). I designed it for creating hard-links in my media collection, but in the --dry-run mode, it should emit the file-names allowing you to pass it to xargs to remove them if it looks copacetic.

You can use my dedupe.py script with the dry-run flag (-n) to find all the duplicates on your drive. If you run it without the dry-run flag, it will attempt to make hard-links so that each file exists only once on the drive with multiple hard-links to the underlying file. It should be pretty fast, only needing to checksum file-content in the event that files have the same size (several other such deduplication methods work by checksumming every file on the drive which can be slow).

a file-deduplication utility that hard-links duplicate files to save space (our family photo gallery gets pics put in multiple albums for various audiences, so I can cut down on a lot of duplication with this)

I use a lot of CLI tools, but haven't written many for myself. Mostly, aliases/functions and some scripts in Bash/Python.

Extract details for command options from man/help: https://github.com/learnbyexample/command_help/blob/master/c...

cut-like syntax for field manipulations with regexp, negative indexing, etc: https://github.com/learnbyexample/regexp-cut/blob/main/rcut

simple calculator using python syntax: https://learnbyexample.github.io/practice_python_projects/ca...

I wrote a Linux CLI tool [0] that parses the man/help pages to extract option details. Works most of the time for me, but there are plenty of corner cases that don't work.

[0] https://github.com/learnbyexample/command_help

https://github.com/learnbyexample/command_help to extract help text from builtin commands and man pages, ex:

https://github.com/learnbyexample/command_help is big enough to warrant a repo, examples, limitations, etc. I had a list of todo items to improve the script, but after years of usage, I'm fine with the limitations since I rarely encounter them. This helps me to extract documentation of particular options, here's an example:

I wrote one a few years back (https://github.com/learnbyexample/command_help). It has a few corner case issues, but works most of the time for me and supports multiple options to be retrieved.

I particularly wanted to lookup documentation for command options from my terminal (instead of the website), so wrote a script for it: https://github.com/learnbyexample/command_help ... Have a long pending todo list, but despite the issues, the tool is good enough for my needs.

When I start a project, I usually try to write the programs incrementally. Say I need to iterate over files from a directory. I will make sure that portion is working (usually with print() statements), then add another feature — say file reading and test that and so on. This reduces the burden of testing a large program at once at the end. And depending upon the nature of the program, I'll add a few sanity tests at the end. For example, for my command_help project, I copy pasted a few test runs of the program with different options and arguments into a separate file and wrote a program to perform these tests programmatically whenever the source code is modified.