beir VS tantivy

The BEIR project might be what you're looking for: https://github.com/beir-cellar/beir/wiki/Leaderboard

Custom datasets can also be evaluated using this method as specified in this link. This article and the associated benchmarks script can be reused to evaluate what method works best on your data.

> Meilisearch focuses on simplicity, relevancy, and performance.

> excellent relevance out of the box

> if ease of use, performance, and relevancy are important to you, Meilisearch was made for you

Is there a benchmark that shows Meilisearch outperforming Elasticsearch in terms of relevance score? I couldn't find Meilisearch listed on https://github.com/beir-cellar/beir.

There are actually benchmarks that allow measuring search relevancy objectively, e.g. BEIR[1]. Manticore Search team did an effort to make a PR to include it to the list. The results are here [2]. Unfortunately the BEIR team seems to be too busy to review a whole pile of PRs including about Vespa. Nevertheless it would be nice to have both Meilisearch and Typesense there too since it's interesting what performance those non-tf-idf based search engines would show compared to BM25-based and vector search engines.

[1] https://github.com/beir-cellar/beir

True! Here's a pull request to BEIR to compare Manticore with Elasticsearch in terms of relevance https://github.com/beir-cellar/beir/pull/92. Spoiler: in this test Manticore provides better relevance than Elasticsearch in average. Of course you can tune both further and Elasticsearch now has KNN which when combined with BM25 can give even better relevance. In general I would say for most users the results quality in terms of full-text relevance is about the same in Elasticseach and Manticore.

But there's for example BEIR that compared BM25 vs state of the art ML language models and it turned out BM25 is in average better than all of them unless you rerank top 100 results from Elasticsearch using the language models. With Manticore you can get even better relevance than with Elasticsearch. We made a pull-request to BEIR to demonstrate that https://docs.google.com/spreadsheets/d/1_ZyYkPJ_K0st9FJBrjbZqX14nmCCPVlE_y3a_y5KkYI/edit#gid=0

Tantivy - a full-text indexing library written in Rust. Has a great performance and featureset.

By this I presume you mean build a search index that can retrieve results based on keywords? I know certain databases use Lucene to build a keyword-based index on top of unstructured blobs of data. Another alternative is to use Tantivy (https://github.com/quickwit-oss/tantivy), a Rust version of Lucene, if building search indices via Java isn't your cup of tea :)

Both libraries offer multilingual support for keywords, I believe, so that's a benefit to vector search where multilingual embedding models are rather expensive.

We also implemented our schemaless columnar storage optimized for object storage.

The inverted index and columnar storage are part of tantivy [0], which is the fastest search library out there. We maintain it and we decided to build the distributed engine on top of it.

[0] tantivy github repo: https://github.com/quickwit-oss/tantivy

The issue for geo search is here: https://github.com/quickwit-oss/tantivy/issues/44

Search index : Custom-built using tantivy.

The roaring bitmap variant is used only for the optional index (1 docid => 0 or 1 value) in the columnar storage (DocValues), not for the inverted index. Since this is used for aggregation, some queries may be a full scan.

The inverted index in tantivy uses bitpacked values of 128 elements with a skip index on top.

> I didn't follow the rest of your comment, select is what EF is good at, every other data structure needs a lot more scanning once you land on the right chunk. With BMI2 you can also use the PDEP instruction to accelerate the final select on a 64-bit block

The select for the sparse codec is a [simple array index access](https://github.com/quickwit-oss/tantivy/blob/main/columnar/s...), that is hard to beat. Compression is not good near the 5k threshold though.

Hi /r/rust, I’m a SWE on Etsy’s Retrieval Systems team where we’re building a platform based on rust and tantivy (https://github.com/quickwit-oss/tantivy). We’re looking to bring two new engineers onto the team.

Quick Search: Easily find specific notes with Velo's fuzzy-search feature, powered by tantivy. tantivy might have been a little overkill, but it was really easy to integrate.