redb VS qdrant

I wrote redb (https://github.com/cberner/redb) using mmap, initially. However, I later removed it and switched to read()/write() with my own user space cache. I'm sure it's not as good as the OS page cache, but the difference was only 1.2-1.5x performance on the benchmarks I cared about, and the cache is less than 500 lines of code.

exposure of the redb builder functionalities.

redb has reached its 1.0 release. The file format is now gauranteed to be backward compatible, and the API is stable. I've run pretty extensive fuzz testing, but please report any bugs you encounter.

I'm just now updating the benchmarks in the readme with multi-threaded workloads :) https://github.com/cberner/redb/pull/576

redb, but that doesn't have a stable file format yet,

I also have to support web, so I'm considering a hybrid approach with a direct IndexedDB API there and something like sqlite, percy, sanakirja, or redb on native.

Nice list, but don’t forget persy, redb and sanakirja too!

check out https://github.com/cberner/redb! it is inspired by LMDB and has comparable performance, but is entirely built in rust!

Yup, the language is immature in some domains like data replication, compilation time, lifetime expressivity, not having a default async runtime and the incompatibility of all of them. But many of these domains are worked on and improved every day this is why I love this language so much. This is because there is space for improvement. I also hope that one day we will have a pure-Rust LMDB alternative that I can contribute to without fear of C.

they are, ya! I'm quite excited about that, and already have a PR open to use GATs. It's not enough to fix this issue though, because std needs to add support for LendingIterator too

Great. Now that we have the embeddings, we need to store them in a vector database. We will be using Qdrant for this purpose. Qdrant is an open-source vector database that allows you to store and query high-dimensional vectors. The easiest way to get started with the Qdrant database is using the docker.

I took Qdrant for this project. The reason was that Qdrant stands for high-performance vector search, the best choice against use cases like finding similar function calls based on semantic similarity. Qdrant is not only powerful but also scalable to support a variety of advanced search features that are greatly useful to nuanced caching mechanisms like ours.

I'm currently looking to implement locally, using QDrant [1] for instance.

I'm just playing around, but it makes sense to have a runnable example for our users at work too :) [2].

[1]. https://qdrant.tech/

Also compare with qdrant's Rust implementation; they tout their performance. https://github.com/qdrant/qdrant/tree/master/lib/segment/src...

There are much better known examples, such as https://qdrant.tech/ and https://github.com/lancedb/lancedb

For more information, see our release notes. Qdrant is an open source project. We welcome your contributions; raise issues, or contribute via pull requests!

Initialize the Qdrant Client with in-memory storage. The collection name will be “imagebind_data” and we will be using cosine distance.

Qdrant is an open-source vector search engine optimized for performance and flexibility. It supports both exact and approximate nearest neighbor search, providing a balance between accuracy and speed for various AI and ML applications.