tantivy VS pest

Backend uses tantivy to index the web pages, sqlite3 to hold metadata / crawl queue

Tantivy is a very well made library and also follows alot of the best practices if you like search you'll like this: https://github.com/quickwit-inc/tantivy

tantivy - More of a search engine library than out of the box solution

Tantivy search engine.

You could try the Tantivy crate, with an n-gram tokenizer, which would split and index your text in sliding groups of n characters.

I had to look up the "tantivy" that README mentions. https://github.com/tantivy-search/tantivy. Might want to add a link to the project in your README.

We're using Rust for a domain-specific search engine. When I first learned Rust some years ago my first thought was that this language is perfect for heavy text processing. IMO, &str is that single killer feature that got me sold :) The search engine that we're building is based on https://github.com/tantivy-search/tantivy.

Well spotted. Like IPFS, there's a comment about that here: https://github.com/tantivy-search/tantivy/pull/1067#issuecomment-853139923 that points to the distributed wikipedia mirror project https://github.com/ipfs/distributed-wikipedia-mirror/issues/76

And some related parser tools: - https://github.com/kevinmehall/rust-peg - https://github.com/pest-parser/pest - https://github.com/lalrpop/lalrpop

I had recently completed the first year of my Computer Science class at school and will begin my second year soon. My schools' class forces the use of Java programming language, and I absolutely hated it. So, over the course of a little less than a month, I wrote my own programming language, in Rust (objectively best programming language), using pest, to be as similar to Rust as possible, but compiling to Java.

I second pest.rs. Using it is fairly intuitive and there's also a live playground on their website which is great for quickly developing and testing your AST (abstract syntax tree) parser for whatever language you're implementing.

This is where you'll run into trouble. People who write parsing-related Rust crates generally write things like pest that expect their syntax to be defined completely at compile time so the parser can be run through the compiler's optimizers for best performance.

Give https://pest.rs a try.

For now I have used pest to generate an AST that borrows from the given input. But if I can manage to make the parser generic over the return type it may be worth a refactoring.