milli VS sage

Is this a posting list? There is a lot of bioinformatics in this post, but if I squint, some of the problems do look like general information retrieval problems. Even the discussion of ordering the arrays by mass sounds like search relevance scores and makes me wonder if it makes sense to try to get something off the shelf like meillisearch/milli or tantavy to support this use case.

I'm currently not using any tool, although I am playing around with binding Milli and the most up-to-date Mentat fork to emacs with emacs-module-rs.

Yeah, we have already done that, the internal engine is called milli and could even be published on crates.io one day! The issue is with the design of the storage system itself, we use LMDB right now but maybe we can find another way to index faster and to be more oriented to distributed systems.

They have another prototype engine with more advanced features and performance too.

https://github.com/meilisearch/milli

You can look at the milli repository this is the library that we use and work on. MeiliSearch is the HTTP actix-web based server that serves the milli indices.

This library is the main bottleneck of the new MeiliSearch search engine. We will soon release a beta version, keep watching!

Working on the new MeiliSearch engine, reworked from scratch! There already is excellent external contributions 🎉

If you want to implement LDA from scratch, you could check out how Sage is doing it.

You could check out https://github.com/lazear/sage - it's a near comprehensive program/pipeline for analyzing DDA/shotgun proteomics data. Most proteomics pipelines consist of running multiple, separate tools in sequence (search, spectrum rescoring, retention time prediction, quantification), but sage performs all of these. This cuts down on the need for disk space for storing intermediate results (none required), the need for IO (files are read once), and results in a proteomics pipeline that is >10-1000x faster than anything else, including commercial solutions

You can also check out the intro blog post if you're interesting in learning more about the algorithm behind Sage. Beyond being fast, it also includes integrated machine learning (linear discriminant analysis, KDE) for rescoring spectral matches.

You could try out Sage, if you're looking for speed - I don't think you'll find anything faster. https://github.com/lazear/sage