tikv VS MeiliSearch

I know that very similar functionality to this is in TiDB Serverless ( https://tidbcloud.com ). TiDB is a distributed relational database. It uses TiKV ( which is a key/value engine ) as the storage engine. You could use SQL to access your K/V records. There is ongoing work in TiKV to support S3 directly as the storage backend ( https://github.com/tikv/tikv/issues/6506 ) .

TiKV is basically a layer on top of rocksdb https://github.com/tikv/tikv/blob/956610725039835557e7516828...

> Many,many smart people…

If you look inside the code you can see the stated features are a result of underlying engine (TiKV [0] also in c and rust from pingcap). Surrealdb is standing on shoulders of giants at present, they are TiKV, FoundationDB and rocksdb. The feature set they mentioned mostly coming from TiKV at present.

[0] https://tikv.org/

Hi Diggsey, great question. We are currently focussed on functionality and stability, and then will draw our attention to performance. Coming this week we have a RocksDB storage implementation. We've only just launched our initial beta version, and we know there is a lot of improvement and work to be done (some of these performance issues we know about already and are on our Github issues list).

With regards to the consistency/isolation model, SurrealDB sits on top of a number of key-value stores. By using the distributed highly-available TiKV storage backend, https://tikv.org, (and we have a FoundationDB integration in the works), the database is designed to be highly-scalable and highly-available. The same guarantees (albeit just single-node, so no high-availability or scalability) will be available with the RocksDB implementation coming this week. By sitting on top of these key-value stores, SurrealDB ensures that all transactions are ACID compliant. We don't want to go for speed (for instance by writing to /dev/null) over anything, but want SurrealDB to be a reliable and performant backend for any application. Obviously we have a way to go to catch up with PostgreSQL (launched in 1996), but we will strive to get there!

If you are looking for a production ready distributed store written in Rust. Check out TiKV(https://github.com/tikv/tikv), which was also mentioned in the acknowledge section of the project's README.

There's also a full-featured distributed RDBMS called TiDB built on top of TiKV.

We needed to introduce a new service for search. As we settled on using meilisearch, we needed a way to sync updates on our models with the records in meilisearch. We could've continued to use callbacks but we needed something better.

Meilisearch

In this case, a good strategy is to use vector search only when the keyword/prefix search returns none or just a small number of results. A good candidate for this is MeiliSearch. It uses custom ranking rules to provide results as fast as the user can type.

With the VertexAI endpoint set up and tested, our next step is to work with LibreChat. LibreChat is an open-source ChatGPT clone that can integrate with various AI models, including the PaLM 2 models via the VertexAI API. It's built using React, MongoDB, and Meilisearch technologies.

Meilisearch seems like it is the best open source option.

https://www.meilisearch.com/

Starting with v1.3, you can use Meilisearch as a vector store. Meilisearch allows you to store vector embeddings alongside your documents conveniently. You will need to create the vector embeddings using your third-party tool of choice (Hugging Face, OpenAI). As we published the first v1.3 release candidate, you can try out vector search today.

I work at Meilisearch, an open-source search engine built in Rust. 🦀

https://www.meilisearch.com/ https://github.com/meilisearch