pipr VS tantivy

You're probably looking for pipr.

I'd highly reccomended trying the program pipr , it shows a live bash output and is helpful for learning what various commands do (although this program has some protection from cmds like rm erasing entire major directories, if you do use it just be careful :-) )

sad CLI search and replace | Space Age seD tcount Count your code by tokens, types of syntax tree nodes, and patterns in the syntax tree. A tokei/scc/cloc alternative. nushell A new type of shell fclones Efficient Duplicate File Finder hunter The fastest file manager in the galaxy! teip Select partial standard input and replace with the result of another command efficiently cb Command line interface to manage clipboard semiuniq A uniq-like tool for removing nearby repeated lines in a file" dua-cli View disk space usage and delete unwanted data, fast. htmlq Like jq, but for HTML. pipecolor A terminal filter to colorize output crowbook Converts books written in Markdown to HTML, LaTeX/PDF and EPUB delta A viewer for git and diff output mdcat cat for markdown pueue Manage your shell commands. gitui Blazing 💥 fast terminal-ui for git written in rust 🦀 pipr A tool to interactively write shell pipelines. rename Rename your files using your favorite text editor bropages Highly readable supplement to man pages from http://bropages.org. Shows simple, concise examples for commands with syntax highlighting. html2md convert simple html documents into markdown bk Terminal Epub reader rs A safe Rust crate for working with the Wayland clipboard. viu Simple terminal image viewer written in Rust. alacritty A cross-platform, OpenGL terminal emulator. wezterm A GPU-accelerated cross-platform terminal emulator and multiplexer written by @wez and implemented in Rust

Backend uses tantivy to index the web pages, sqlite3 to hold metadata / crawl queue

Tantivy is a very well made library and also follows alot of the best practices if you like search you'll like this: https://github.com/quickwit-inc/tantivy

tantivy - More of a search engine library than out of the box solution

Tantivy search engine.

You could try the Tantivy crate, with an n-gram tokenizer, which would split and index your text in sliding groups of n characters.

I had to look up the "tantivy" that README mentions. https://github.com/tantivy-search/tantivy. Might want to add a link to the project in your README.

We're using Rust for a domain-specific search engine. When I first learned Rust some years ago my first thought was that this language is perfect for heavy text processing. IMO, &str is that single killer feature that got me sold :) The search engine that we're building is based on https://github.com/tantivy-search/tantivy.

Well spotted. Like IPFS, there's a comment about that here: https://github.com/tantivy-search/tantivy/pull/1067#issuecomment-853139923 that points to the distributed wikipedia mirror project https://github.com/ipfs/distributed-wikipedia-mirror/issues/76