dedupe
ripgrep-all
dedupe | ripgrep-all | |
---|---|---|
3 | 43 | |
3 | 6,177 | |
- | - | |
0.0 | 8.0 | |
about 6 years ago | 2 months ago | |
Python | Rust | |
BSD 2-clause "Simplified" License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
dedupe
-
fdupes alternatives?
I wrote https://github.com/Gumnos/dedupe which sounds like it might be useful to you. It's faster than several of the alternatives I've found (many run the checksum across the whole of every file, this uses the file-size as a first-line discriminator, and only if the files are the same size does it go to the trouble of checking the checksum of the files). I designed it for creating hard-links in my media collection, but in the --dry-run mode, it should emit the file-names allowing you to pass it to xargs to remove them if it looks copacetic.
-
File Management via CLI
You can use my dedupe.py script with the dry-run flag (-n) to find all the duplicates on your drive. If you run it without the dry-run flag, it will attempt to make hard-links so that each file exists only once on the drive with multiple hard-links to the underlying file. It should be pretty fast, only needing to checksum file-content in the event that files have the same size (several other such deduplication methods work by checksumming every file on the drive which can be slow).
-
What tools / utilities have you written that you use regularly?
a file-deduplication utility that hard-links duplicate files to save space (our family photo gallery gets pics put in multiple albums for various audiences, so I can cut down on a lot of duplication with this)
ripgrep-all
- Ripgrep-all: rga: ripgrep, but also search PDFs, E-Books, Office documents, zip
-
Ripgrep is faster than {grep, ag, Git grep, ucg, pt, sift}
I searched in portage, and it seems there is another version working also with other documents like PDFs and doc.
https://github.com/phiresky/ripgrep-all
-
Calibre – New in Calibre 7.0
If you want even faster search across different formats, you can try ripgrep-all ( https://github.com/phiresky/ripgrep-all ). It can search across epub, docx, pdf, zip, mp4 etc. If you are handy with the tool, you can write custom adaptor to search across images using OCR with tesseract.
- Rga: Ripgrep, but also search in PDF, ebooks, office documents, zip, tar.gz etc.
-
Show HN: Khoj – Chat Offline with Your Second Brain Using Llama 2
1. If you want better adoption especially among corporations, GPL-3 wont cut it. Maybe think of some business friendly licenses (MIT etc)
2. I understand the excitement about llm's. But how about making something more accessible. I use rip-grep-all (rga) along with fzf [1] that can search all files including pdfs in a specific folders. However, I would like a GUI tool to search across multiple folders, provide priority of results across folders and store and search histories where I can do a meta-search. This is sufficient for 95% of my usecases to search locally and I dont need LLM. If khoj can enable such search as default without LLM that will be a gamechanger for many people without a heavy compute machine or who dont want to use OpenAI.
[1] https://github.com/phiresky/ripgrep-all/wiki/fzf-Integration
-
How to make file paths clickable?
I use `rga` to search through multiple PDF files for work. The tool returns a list of files and I would like to make those file paths clickable.
- Burgr – Books in Your Terminal
-
Is there a way to searching multiple epub and pdf?
rga, aka ripgrep-all
-
Internet Archive Scholar
I wanted to say 'au contrer' to your 'screenshots are not searchable' and link this[0] but I don't actually see images in the readme.. I swear it was there, maybe it's a buried extra flag..
[0] https://github.com/phiresky/ripgrep-all
- Recoll – Full-text search for your desktop
What are some alternatives?
file-arranger - Simple & capable Directory arranger/cleaner
pdfgrep - PDFGrep is a GNU/Emacs module providing grep comparable facilities but for PDF files
xonsh - :shell: Python-powered, cross-platform, Unix-gazing shell.
OCRmyPDF - OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to be searched
tawk - Like awk, but using tcl as the scripting language.
InvoiceNet - Deep neural network to extract intelligent information from invoice documents.
mpd_what - An mpd album art and info getter
notational-fzf-vim - Notational velocity for vim.
ledger - Double-entry accounting system with a command-line reporting interface
fd - A simple, fast and user-friendly alternative to 'find'
fzf - :cherry_blossom: A command-line fuzzy finder
ripgrep - ripgrep recursively searches directories for a regex pattern while respecting your gitignore