Vector Search Indexes

• pgvector

  78 9,211 9.9 C

  Open-source vector similarity search for Postgres

  

Apart from search, ANN could be use for recommendations, classification, and other information retrieval problems.

Currently, ES and Solr, both based on Lucene, can't really manage vector representations, as they are mainly based on inverted indexes to n-grams.

ANN potential applications extend to audio, bioinformatics, video, among any modality that can be represented as a vector. All you need is an encoder! How nice.

Faiss is definitely powerful. I have been running experiments using 80 million vectors that map to legal documents, and vectorizing protein-folds (using Alphafold). While it is an interesting technology, at this moment is more a lib or tool than a full-featured product like ES or Solr.

ATM, updating a Faiss index is a non trivial process, with many of the workflow tools you would see in ES missing.

I recently been found with pgvector (postgres + vector support) https://github.com/ankane/pgvector. Perhaps less performant, but easier to work with for teams. With support of migrations, ORM, sharing, and all the postgres goodies.

• Weaviate

  76 9,524 10.0 Go

  Weaviate is an open-source vector database that stores both objects and vectors, allowing for the combination of vector search with structured filtering with the fault tolerance and scalability of a cloud-native database​.

  

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• vald

  13 1,455 9.3 Go

  Vald. A Highly Scalable Distributed Vector Search Engine

  

• Milvus

  104 26,857 10.0 Go

  A cloud-native vector database, storage for next generation AI applications

  

• nmslib

  4 3,278 0.0 C++

  Non-Metric Space Library (NMSLIB): An efficient similarity search library and a toolkit for evaluation of k-NN methods for generic non-metric spaces.

  

nmslib (https://github.com/nmslib/nmslib) supports sparse vectors for some of its spaces. It has fewer indexing methods than faiss, though.

https://github.com/nmslib/nmslib/blob/master/manual/spaces.m...

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project