Awesome-Rust-MachineLearning VS tantivy

There’s an awesome-git list for a bunch of ML rust stuff not sure how up to date it is as well https://github.com/vaaaaanquish/Awesome-Rust-MachineLearning … not mine

I am looking for something like [Triton Inference Server](https://github.com/triton-inference-server/server) or [TFX Serving](https://www.tensorflow.org/tfx/guide/serving), but in Rust. I came across [Orkon](https://github.com/vertexclique/orkhon) which seems to be dormant and a bunch of examples off of the [Awesome-Rust-MachineLearning](https://github.com/vaaaaanquish/Awesome-Rust-MachineLearning)

There's also https://github.com/vaaaaanquish/Awesome-Rust-MachineLearning

I saw Awesome-Rust-MachineLearning. It have something replacements from Python.

Backend uses tantivy to index the web pages, sqlite3 to hold metadata / crawl queue

Tantivy is a very well made library and also follows alot of the best practices if you like search you'll like this: https://github.com/quickwit-inc/tantivy

tantivy - More of a search engine library than out of the box solution

Tantivy search engine.

You could try the Tantivy crate, with an n-gram tokenizer, which would split and index your text in sliding groups of n characters.

I had to look up the "tantivy" that README mentions. https://github.com/tantivy-search/tantivy. Might want to add a link to the project in your README.

We're using Rust for a domain-specific search engine. When I first learned Rust some years ago my first thought was that this language is perfect for heavy text processing. IMO, &str is that single killer feature that got me sold :) The search engine that we're building is based on https://github.com/tantivy-search/tantivy.

Well spotted. Like IPFS, there's a comment about that here: https://github.com/tantivy-search/tantivy/pull/1067#issuecomment-853139923 that points to the distributed wikipedia mirror project https://github.com/ipfs/distributed-wikipedia-mirror/issues/76