Vector Search Indexes

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • OPS - Build and Run Open Source Unikernels
  • Scout APM - Less time debugging, more time building
  • SonarLint - Deliver Cleaner and Safer Code - Right in Your IDE of Choice!
  • GitHub repo pgvector

    Open-source vector similarity search for Postgres

    Apart from search, ANN could be use for recommendations, classification, and other information retrieval problems.

    Currently, ES and Solr, both based on Lucene, can't really manage vector representations, as they are mainly based on inverted indexes to n-grams.

    ANN potential applications extend to audio, bioinformatics, video, among any modality that can be represented as a vector. All you need is an encoder! How nice.

    Faiss is definitely powerful. I have been running experiments using 80 million vectors that map to legal documents, and vectorizing protein-folds (using Alphafold). While it is an interesting technology, at this moment is more a lib or tool than a full-featured product like ES or Solr.

    ATM, updating a Faiss index is a non trivial process, with many of the workflow tools you would see in ES missing.

    I recently been found with pgvector (postgres + vector support) https://github.com/ankane/pgvector. Perhaps less performant, but easier to work with for teams. With support of migrations, ORM, sharing, and all the postgres goodies.

  • GitHub repo Weaviate

    Weaviate is a cloud-native, modular, real-time vector search engine

  • OPS

    OPS - Build and Run Open Source Unikernels. Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.

  • GitHub repo vald

    Vald. A Highly Scalable Distributed Vector Search Engine

  • GitHub repo milvus

    An open-source vector database for embedding similarity search and AI applications.

  • GitHub repo nmslib

    Non-Metric Space Library (NMSLIB): An efficient similarity search library and a toolkit for evaluation of k-NN methods for generic non-metric spaces.

    nmslib (https://github.com/nmslib/nmslib) supports sparse vectors for some of its spaces. It has fewer indexing methods than faiss, though.

    https://github.com/nmslib/nmslib/blob/master/manual/spaces.m...

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts