VDBforGenAI
pgvector
VDBforGenAI | pgvector | |
---|---|---|
1 | 82 | |
0 | 10,077 | |
- | 8.6% | |
6.1 | 9.9 | |
12 months ago | 5 days ago | |
Python | C | |
MIT License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
VDBforGenAI
-
[D] Simplest Vector DB Implementation?
I made this one because I was asking a similar question some time ago. That is find the nearest vector from a bunch of documents and do it quickly and efficiently. Faiss backend with some nice python utilities. https://github.com/JakubJDolezal/VDBforGenAI
pgvector
-
SpringAI, llama3 and pgvector: bRAGging rights!
To support the exploration, I've developed a simple Retrieval Augmented Generation (RAG) workflow that works completely locally on the laptop for free. If you're interested, you can find the code itself here. Basically, I've used Testcontainers to create a Postgres database container with the pgvector extension to store text embeddings and an open source LLM with which I send requests to: Meta's llama3 through ollama.
- Open-source vector similarity search for Postgres
-
beginner guide to fully local RAG on entry-level machines
# postgres image with `pgvector` enabled FROM postgres:16.3 RUN apt-get update \ && apt-get install -y postgresql-server-dev-all build-essential \ && apt-get install -y git \ && git clone https://github.com/pgvector/pgvector.git \ && cd pgvector \ && make \ && make install \ && apt-get remove -y git build-essential \ && apt-get autoremove -y \ && rm -rf /var/lib/apt/lists/* EXPOSE 5432
-
Show HN: Smart website search powered by open models
I helped work on the RAG part of this :-)
We used https://github.com/pgvector/pgvector under the hood and found it extremely easy to integrate with our database schema - being able to just specify the structure of a table and have metadata fields alongside the embeddings made the code very easy to reason about.
-
Integrate txtai with Postgres
# Install Postgres and pgvector !apt-get update && apt install postgresql postgresql-server-dev-14 !git clone --branch v0.6.2 https://github.com/pgvector/pgvector.git !cd pgvector && make && make install # Start database !service postgresql start !sudo -u postgres psql -U postgres -c "ALTER USER postgres PASSWORD 'pass';"
-
Vector Database solutions on AWS
When talking about Vector Databases, in the market we can find the specialized ones and multi-model, most of the major database providers like Oracle, PostgreSQL or MongoDB, for mention some of them, have integrated a specific solution to retrieve vector data.
-
Using pgvector To Locate Similarities In Enterprise Data
For this example, I wanted to focus on how pgvector โ an open-source vector similarity search for Postgres โ can be used to identify data similarities that exist in enterprise data.
-
pgvector vs. pgvecto.rs in 2024: A Comprehensive Comparison for Vector Search in PostgreSQL
pgvector supports dense vector search well, but it does not have plan to support sparse vector.
-
Pg_vectorize: The simplest way to do vector search and RAG on Postgres
There's an issue in the pgvector repo about someone having several ~10-20million row tables and getting acceptable performance with the right hardware and some performance tuning: https://github.com/pgvector/pgvector/issues/455
I'm in the early stages of evaluating pgvector myself. but having used pinecone I currently am liking pgvector better because of it being open source. The indexing algorithm is clear, one can understand and modify the parameters. Furthermore the database is postgresql, not a proprietary document store. When the other data in the problem is stored relationally, it is very convenient to have the vectors stored like this as well. And postgresql has good observability and metrics. I think when it comes to flexibility for specialized applications, pgvector seems like the clear winner. But I can definitely see pinecone's appeal if vector search is not a core component of the problem/business, as it is very easy to use and scales very easily
- FLaNK 04 March 2024
What are some alternatives?
usearch - Fast Open-Source Search & Clustering engine ร for Vectors & ๐ Strings ร in C++, C, Python, JavaScript, Rust, Java, Objective-C, Swift, C#, GoLang, and Wolfram ๐
Milvus - A cloud-native vector database, storage for next generation AI applications
faiss - A library for efficient similarity search and clustering of dense vectors.
Weaviate - Weaviate is an open-source vector database that stores both objects and vectors, allowing for the combination of vector search with structured filtering with the fault tolerance and scalability of a cloud-native databaseโ.
Elasticsearch - Free and Open, Distributed, RESTful Search Engine
qdrant - Qdrant - High-performance, massive-scale Vector Database for the next generation of AI. Also available in the cloud https://cloud.qdrant.io/
ann-benchmarks - Benchmarks of approximate nearest neighbor libraries in Python
pinecone - Peer-to-peer overlay routing for the Matrix ecosystem
smlar - PostgreSQL extension for an effective similarity search || mirror of git://sigaev.ru/smlar.git || see https://www.pgcon.org/2012/schedule/track/Hacking/443.en.html
vespa - AI + Data, online. https://vespa.ai
awesome-vector-search - Collections of vector search related libraries, service and research papers
patroni - A template for PostgreSQL High Availability with Etcd, Consul, ZooKeeper, or Kubernetes