pgvector
faiss
Our great sponsors
pgvector | faiss | |
---|---|---|
77 | 70 | |
8,904 | 27,924 | |
7.3% | 3.4% | |
9.7 | 9.4 | |
6 days ago | 5 days ago | |
C | C++ | |
GNU General Public License v3.0 or later | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
pgvector
-
Using pgvector To Locate Similarities In Enterprise Data
For this example, I wanted to focus on how pgvector – an open-source vector similarity search for Postgres – can be used to identify data similarities that exist in enterprise data.
-
Pg_vectorize: The simplest way to do vector search and RAG on Postgres
There's an issue in the pgvector repo about someone having several ~10-20million row tables and getting acceptable performance with the right hardware and some performance tuning: https://github.com/pgvector/pgvector/issues/455
I'm in the early stages of evaluating pgvector myself. but having used pinecone I currently am liking pgvector better because of it being open source. The indexing algorithm is clear, one can understand and modify the parameters. Furthermore the database is postgresql, not a proprietary document store. When the other data in the problem is stored relationally, it is very convenient to have the vectors stored like this as well. And postgresql has good observability and metrics. I think when it comes to flexibility for specialized applications, pgvector seems like the clear winner. But I can definitely see pinecone's appeal if vector search is not a core component of the problem/business, as it is very easy to use and scales very easily
- FLaNK 04 March 2024
-
Vector Database and Spring IA
The Spring AI project aims to streamline the development of applications that incorporate artificial intelligence functionality without unnecessary complexity. On this example we use features like: Embedding, Prompts, ETL and save all embedding on PGvector(Postgres Vector database)
-
Use pgvector for searching images on Azure Cosmos DB for PostgreSQL
Official GitHub repository of the pgvector extension
-
Store embeddings in Azure Cosmos DB for PostgreSQL with pgvector
The pgvector extension adds vector similarity search capabilities to your PostgreSQL database. To use the extension, you have to first create it in your database. You can install the extension, by connecting to your database and running the CREATE EXTENSION command from the psql command prompt:
-
Are we at peak vector database?
It’s about to get a lot better too. Pgvector now supports multi-threaded build
https://github.com/pgvector/pgvector/issues/409#issuecomment...
pgvector has you covered: https://github.com/pgvector/pgvector
-
Vector Databases: A Technical Primer [pdf]
You don't need a dedicated vector db, you can use pgvector.
You could maybe use Cube for euclidean space search, but you're better off using optimized algorithms for embedding space search.
-
How Modern SQL Databases Are Changing Web Development - #4 Into the AI Era
PostgreSQL's pgvector extension is probably the most widely used SQL solution for storing and searching vector data today. The extension introduces a "vector" type specialized for storing high-dimensional vector data. It allows you to create vector indices (in "IVFFlat" or "HNSW" format for different indexing/searching performance tradeoffs) and leverage them to do various types of similarity searches.
faiss
-
Show HN: Chromem-go – Embeddable vector database for Go
Or just use FAISS https://github.com/facebookresearch/faiss
-
You Shouldn't Invest in Vector Databases?
You can try txtai (https://github.com/neuml/txtai) with a Faiss backend.
This Faiss wiki article might help (https://github.com/facebookresearch/faiss/wiki/Indexing-1G-v...).
For example, a partial Faiss configuration with 4-bit PQ quantization and only using 5% of the data to train an IVF index is shown below.
faiss={"components": "IVF,PQ384x4fs", "sample": 0.05}
-
Approximate Nearest Neighbors Oh Yeah
If you want to experiment with vector stores, you can do that locally with something like faiss which has good platform support: https://github.com/facebookresearch/faiss
Doing full retrieval-augmented generation (RAG) and getting LLMs to interpret the results has more steps but you get a lot of flexibility, and there's no standard best-practice. When you use a vector DB you get the most similar texts back (or an index integer in the case of faiss), you then feed those to an LLM like a normal prompt.
The codifer for the RAG workflow is LangChain, but their demo is substantially more complex and harder-to-use than even a homegrown implementation: https://news.ycombinator.com/item?id=36725982
-
Can someone please help me with this problem?
According to this documentation page, faiss-gpu is only supported on Linux, not on Windows.
-
Code Search with Vector Embeddings: A Transformer's Approach
As the size of the codebase grows, storing and searching through embeddings in memory becomes inefficient. This is where vector databases come into play. Tools like Milvus, Faiss, and others are designed to handle large-scale vector data and provide efficient similarity search capabilities. I've wrtten about how to also use sqlite to store vector embeddings. By integrating a vector database, you can scale your code search tool to handle much larger codebases without compromising on search speed.
-
Unum: Vector Search engine in a single file
But FAISS has their own version ("FastScan") https://github.com/facebookresearch/faiss/wiki/Fast-accumula...
- Introduction to Vector Similarity Search
-
Vector Databases 101
If you want to go larger you could still use some simple setup in conjunction with faiss, annoy or hnsw.
- I'm an undergraduate data science intern and trying to run kmodes clustering. Did this elbow method to figure out how many clusters to use, but I don't really see an "elbow". Tips on number of clusters?
-
Disrupting the AI Scene with Open Source and Open Innovation
Facebook research for having open source licensed an amazing vector based indexing library
What are some alternatives?
annoy - Approximate Nearest Neighbors in C++/Python optimized for memory usage and loading/saving to disk
Milvus - A cloud-native vector database, storage for next generation AI applications
hnswlib - Header-only C++/python library for fast approximate nearest neighbors
Weaviate - Weaviate is an open-source vector database that stores both objects and vectors, allowing for the combination of vector search with structured filtering with the fault tolerance and scalability of a cloud-native database.
Elasticsearch - Free and Open, Distributed, RESTful Search Engine
qdrant - Qdrant - High-performance, massive-scale Vector Database for the next generation of AI. Also available in the cloud https://cloud.qdrant.io/
hdbscan - A high performance implementation of HDBSCAN clustering.
ann-benchmarks - Benchmarks of approximate nearest neighbor libraries in Python