charabia VS qdrant

I hope to use this to submit a PR to meilisearch/charabia in the future based on the excellent example by benny-n in his pull request adding Hebrew support.

Good to know. If you find some Russian language support improvement we can do, don't hesitate to create an issue on our tokeniser https://github.com/meilisearch/charabia.

Moreover, it's Hacktoberfest. If you want to help us improve the language support, it would be awesome!

The Language support is mainly handled by the tokenizer where you can find specialized Segmenters and Normalizers that target some Language subtilities: https://github.com/meilisearch/charabia

Hi @tommoor

I work at Meilisearch so maybe a biased answer.

We do support other languages than English, it actually depends on what are the languages supported by our tokenizer https://github.com/meilisearch/charabia, so any language that uses whitespace to separate words(including English), Chinese, Japanese, Hebrew and Thai (Japaneze and Thai might work a little bit less)

Hello u/pkeep-go, at MeiliSearch we are implementing with the help of a contributor the CJK language support. We already have the Chinese support with Jieba but a contributor is adding the Korean and Japaneese support with Lindera. For us, it's really important to have people trying and helping us with languages we don't speak. If you want to make any contribution on the tokenization/language support part, we'll be happy to help!

Great. Now that we have the embeddings, we need to store them in a vector database. We will be using Qdrant for this purpose. Qdrant is an open-source vector database that allows you to store and query high-dimensional vectors. The easiest way to get started with the Qdrant database is using the docker.

I took Qdrant for this project. The reason was that Qdrant stands for high-performance vector search, the best choice against use cases like finding similar function calls based on semantic similarity. Qdrant is not only powerful but also scalable to support a variety of advanced search features that are greatly useful to nuanced caching mechanisms like ours.

I'm currently looking to implement locally, using QDrant [1] for instance.

I'm just playing around, but it makes sense to have a runnable example for our users at work too :) [2].

[1]. https://qdrant.tech/

Also compare with qdrant's Rust implementation; they tout their performance. https://github.com/qdrant/qdrant/tree/master/lib/segment/src...

There are much better known examples, such as https://qdrant.tech/ and https://github.com/lancedb/lancedb

For more information, see our release notes. Qdrant is an open source project. We welcome your contributions; raise issues, or contribute via pull requests!

Initialize the Qdrant Client with in-memory storage. The collection name will be “imagebind_data” and we will be using cosine distance.

Qdrant is an open-source vector search engine optimized for performance and flexibility. It supports both exact and approximate nearest neighbor search, providing a balance between accuracy and speed for various AI and ML applications.