nmslib
TorchPQ
| nmslib | TorchPQ | |
|---|---|---|
| 4 | 3 | |
| 3,281 | 203 | |
| 1.3% | - | |
| 0.0 | 3.5 | |
| about 1 month ago | 5 months ago | |
| C++ | Cuda | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
nmslib
- Vector search just got up to 10x faster and vertically scalable
-
The Missing ”WHERE” Clause for Vector Search
Amazon's Opensearch (fork of Elasticsearch) natively supports vector-based approximate KNN (using https://github.com/nmslib/nmslib/) which is integrated with Opensearch's native filtering functionality. Elasticsearch also has similar functionality, but I don't know if their KNN code scales quite as well.
-
Vector Search Indexes
nmslib (https://github.com/nmslib/nmslib) supports sparse vectors for some of its spaces. It has fewer indexing methods than faiss, though.
https://github.com/nmslib/nmslib/blob/master/manual/spaces.m...
-
Are there more practical tools for KNN searches and storing documents/embeddings?
I also needed to build a similar system and I used nmslib, maybe check it out - https://github.com/nmslib/nmslib
TorchPQ
-
[D] Is there an open-source implementation of the Retrieval-Enhanced Transformer (RETRO)?
if you're decided to run nearest neighbor search on GPU, you might wanna give TorchPQTorchPQ a try
-
[P] TorchPQ: Efficient Nearest Neighbor Search and Clustering on GPUs
TorchPQ is a python library for approximate nearest neighbor search on GPUs. It has efficient implementations of IVFPQ algorithm as well as some of its variants (e.g IVFPQ+R). The project is written mostly in python using pytorch library, with some custom CUDA kernels to accelerate clustering, searching and indexing.
-
Are there more practical tools for KNN searches and storing documents/embeddings?
If you're using GPU for search, You can give TorchPQ a try, it's an ANN search library implemented with PyTorch and CUDA. with TorchPQ, you can freely add / remove vectors with IDs, you can save/load them just like a regular PyTorch model. TorchPQ is still in development, and I'd love to hear any suggestion or feedback.
What are some alternatives?
Milvus - A cloud-native vector database, storage for next generation AI applications
faiss - A library for efficient similarity search and clustering of dense vectors.
pgvector - Open-source vector similarity search for Postgres
RETRO-pytorch - Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch
Weaviate - Weaviate is an open-source vector database that stores both objects and vectors, allowing for the combination of vector search with structured filtering with the fault tolerance and scalability of a cloud-native database.
FirstCollisionTimestepRarefiedGasSimulator - This simulator computes all possible intersections for a very small timestep for a particle model
qdrant - Qdrant - High-performance, massive-scale Vector Database for the next generation of AI. Also available in the cloud https://cloud.qdrant.io/
Scalix - Scalix is a data parallel compute library that automatically scales to the available compute resources.
vald - Vald. A Highly Scalable Distributed Vector Search Engine
deepmind-research - This repository contains implementations and illustrative code to accompany DeepMind publications
knowhere - Knowhere is an open-source vector search engine, integrating FAISS, HNSW, etc.
DOKSparse - sparse DOK tensors on GPU, pytorch