fst VS MeiliSearch

I'd point you towards BurntSushi's fst crate: https://github.com/BurntSushi/fst

The fst crate effectively relies on mmap for it to work right. The folks here suggesting you just use the heap might be right, but only if using the heap is actually plausible. If your dictionary is GBs big (an FST might be bigger than available memory), then copying it the heap first would be disastrous.

You'll love what we're working on if you're interested in the implementation of:- Tantivy- Meilisearch- Finite State Transducers

Here's an example of longest prefix matching using a FST which I based my approach on: https://github.com/BurntSushi/fst/pull/104/files

Finite State Transducers

As a result, I have a plan to implement a desktop version from then and I finished today with a beta version. The desktop is based on tauri, and the dictionary index algorithm is FST (it is an awesome index algorithm).

WordBueno’s data is currently derived from Wiktionary. The backend is using Rust’s warp with fst for indexing.

No you don't. I've written multiple programs that load things instantly off the file system via memory maps. See the fst crate[1], for example, which is designed to work with memory maps.

Rust "works badly with memory mapped files" doesn't mean, "Rust can't use memory mapped files." It means, "it is difficult to reconcile Rust's safety story with memory maps." ripgrep for example uses memory maps because they are faster sometimes, and its safety contract[2] is a bit strained. But it works.

[1] - https://github.com/BurntSushi/fst/

[2] - https://docs.rs/grep-searcher/0.1.7/grep_searcher/struct.Mma...

We needed to introduce a new service for search. As we settled on using meilisearch, we needed a way to sync updates on our models with the records in meilisearch. We could've continued to use callbacks but we needed something better.

Meilisearch

In this case, a good strategy is to use vector search only when the keyword/prefix search returns none or just a small number of results. A good candidate for this is MeiliSearch. It uses custom ranking rules to provide results as fast as the user can type.

With the VertexAI endpoint set up and tested, our next step is to work with LibreChat. LibreChat is an open-source ChatGPT clone that can integrate with various AI models, including the PaLM 2 models via the VertexAI API. It's built using React, MongoDB, and Meilisearch technologies.

Meilisearch seems like it is the best open source option.

https://www.meilisearch.com/

Starting with v1.3, you can use Meilisearch as a vector store. Meilisearch allows you to store vector embeddings alongside your documents conveniently. You will need to create the vector embeddings using your third-party tool of choice (Hugging Face, OpenAI). As we published the first v1.3 release candidate, you can try out vector search today.

I work at Meilisearch, an open-source search engine built in Rust. 🦀

https://www.meilisearch.com/ https://github.com/meilisearch