milli VS prost

Is this a posting list? There is a lot of bioinformatics in this post, but if I squint, some of the problems do look like general information retrieval problems. Even the discussion of ordering the arrays by mass sounds like search relevance scores and makes me wonder if it makes sense to try to get something off the shelf like meillisearch/milli or tantavy to support this use case.

I'm currently not using any tool, although I am playing around with binding Milli and the most up-to-date Mentat fork to emacs with emacs-module-rs.

Yeah, we have already done that, the internal engine is called milli and could even be published on crates.io one day! The issue is with the design of the storage system itself, we use LMDB right now but maybe we can find another way to index faster and to be more oriented to distributed systems.

They have another prototype engine with more advanced features and performance too.

https://github.com/meilisearch/milli

You can look at the milli repository this is the library that we use and work on. MeiliSearch is the HTTP actix-web based server that serves the milli indices.

This library is the main bottleneck of the new MeiliSearch search engine. We will soon release a beta version, keep watching!

Working on the new MeiliSearch engine, reworked from scratch! There already is excellent external contributions 🎉

We've been using Rust with protobuf, but had to write lots of conversions from protobuf struct to actual DTO, since prost generates Options for nested types, and does not really support custom types. So I extended prost with the options that we needed. I might aim to also support protoc-gen-validate.

I'm prototyping an alternative protobuf code generator, with the aim to support more configuration and options. Prost does not have any facilities for implementing extensions, so it's a fork with edits in every single component.