similarity VS qdrant

Tensorflow Ranking and Tensorflow similarity (maybe relevant/irrelevant contrastive learning?) look like they could be useful.

There is the metric learning problem to learn a hash for similarity https://github.com/tensorflow/similarity

That said, I don't see many good models available for download on tfhub or huggingface optimized for it, but you can always programmatically modify your images (if you truly mean identical to humans) - change white balance, crop, rotate, select adjacent frames from videos, etc. and optimize a network that is small enough for you to be satisfied and see if that works, as a possible alternative.

Metric learning has great implementations inside Tensorflow Similarity library: https://github.com/tensorflow/similarity Although the documentation is quite bad, but the jupyter notebooks are great.

Just a quick note that TensorFlow Similarity 0.16 is out -- this release beside adding the XMB loss is mostly focus on refactoring and optimizing the core components to ensure everything works smoothly and accurately. Details are in the changelog as usual and a simple pip install -U tensorflow_similarity should just work.

Very happy to announce that as part of the 0.15 release, TensorFlow Similarity now support self-supervised learning using STOA algorithms. To help you get started we included in the release a detailed getting started notebook that you can run in Colab. This notebook shows you how to use SimSiam self-supervised pre-training to almost double the accuracy compared to a model trained from scratch on CIFAR 10.

Github: https://github.com/tensorflow/similarity

Great. Now that we have the embeddings, we need to store them in a vector database. We will be using Qdrant for this purpose. Qdrant is an open-source vector database that allows you to store and query high-dimensional vectors. The easiest way to get started with the Qdrant database is using the docker.

I took Qdrant for this project. The reason was that Qdrant stands for high-performance vector search, the best choice against use cases like finding similar function calls based on semantic similarity. Qdrant is not only powerful but also scalable to support a variety of advanced search features that are greatly useful to nuanced caching mechanisms like ours.

I'm currently looking to implement locally, using QDrant [1] for instance.

I'm just playing around, but it makes sense to have a runnable example for our users at work too :) [2].

[1]. https://qdrant.tech/

Also compare with qdrant's Rust implementation; they tout their performance. https://github.com/qdrant/qdrant/tree/master/lib/segment/src...

There are much better known examples, such as https://qdrant.tech/ and https://github.com/lancedb/lancedb

For more information, see our release notes. Qdrant is an open source project. We welcome your contributions; raise issues, or contribute via pull requests!

Initialize the Qdrant Client with in-memory storage. The collection name will be “imagebind_data” and we will be using cosine distance.

Qdrant is an open-source vector search engine optimized for performance and flexibility. It supports both exact and approximate nearest neighbor search, providing a balance between accuracy and speed for various AI and ML applications.