sqlite-vss VS pgvector

I guess this is an answer to the GitHub issue I opened against SQLite-vss a couple of months ago?

https://github.com/asg017/sqlite-vss/issues/124

Perhaps sqlite-vss? It adds vector searches to sqlite.

https://github.com/asg017/sqlite-vss

Utilizing sqlite-vss to store and query vector embeddings managed by a local SQLite database, Semantify conducts fast, precise vector searches within these embeddings to find and recommend relevant content, ensuring readers are presented with articles that truly match their interests.

Whether you’re navigating through well-known options like SQLite, enriched with the sqlite-vss extension, or exploring other avenues like Chroma, an open-source vector database, selecting the right tool is paramount. This article compares these two choices, guiding you through the pros and cons of each, helping you choose the right tool for storing and querying vector embeddings for your project.

Here is a SQLite extension that uses Faiss under the hood.

https://github.com/asg017/sqlite-vss

Not associated with the project, just love SQLite and find it very useful.

Vector Databases: As your data grows, efficiently searching through millions of vectors can become a challenge. Specialized vector databases like FAISS, Annoy, or Elasticsearch's vector search capabilities can be explored to manage and search through large-scale vector data. Your sentence is grammatically correct. In addition, databases like SQLite and PostgreSQL have extensions, such as sqlite-vss and pgvector, that can be used to store and query vector embeddings, respectively.

I had a go at one of those a few months ago: https://datasette.io/plugins/datasette-faiss

Alex Garcia built a better one here as a SQLite Rust extension: https://github.com/asg017/sqlite-vss

Since the problem domain is that of anomaly detection from constructed request feature embeddings, I wonder if an ANN-search methodology using an embedded database (such as https://github.com/asg017/sqlite-vss or similar) was explored.

As I searched for "sqlite vector plugin" I didn't find any results, before a couple of weeks ago. Two weeks ago I found Alex' SQLite VSS plugin for SQLite. The library was an amazing piece of engineering from an "idea perspective". However, as I started playing around with it, I realised it was ipso facto like "Titanic". Beautiful and amazing, but destined to leak water and sink to the bottom of the ocean because of what we software engineers refers to as "memory leaks".

# Install Postgres and pgvector !apt-get update && apt install postgresql postgresql-server-dev-14 !git clone --branch v0.6.2 https://github.com/pgvector/pgvector.git !cd pgvector && make && make install # Start database !service postgresql start !sudo -u postgres psql -U postgres -c "ALTER USER postgres PASSWORD 'pass';"

When talking about Vector Databases, in the market we can find the specialized ones and multi-model, most of the major database providers like Oracle, PostgreSQL or MongoDB, for mention some of them, have integrated a specific solution to retrieve vector data.

For this example, I wanted to focus on how pgvector  – an open-source vector similarity search for Postgres – can be used to identify data similarities that exist in enterprise data.

pgvector supports dense vector search well, but it does not have plan to support sparse vector.

There's an issue in the pgvector repo about someone having several ~10-20million row tables and getting acceptable performance with the right hardware and some performance tuning: https://github.com/pgvector/pgvector/issues/455

I'm in the early stages of evaluating pgvector myself. but having used pinecone I currently am liking pgvector better because of it being open source. The indexing algorithm is clear, one can understand and modify the parameters. Furthermore the database is postgresql, not a proprietary document store. When the other data in the problem is stored relationally, it is very convenient to have the vectors stored like this as well. And postgresql has good observability and metrics. I think when it comes to flexibility for specialized applications, pgvector seems like the clear winner. But I can definitely see pinecone's appeal if vector search is not a core component of the problem/business, as it is very easy to use and scales very easily

The Spring AI project aims to streamline the development of applications that incorporate artificial intelligence functionality without unnecessary complexity. On this example we use features like: Embedding, Prompts, ETL and save all embedding on PGvector(Postgres Vector database)

Official GitHub repository of the pgvector extension

pgvector 0.6.0 was just released and will be available on Supabase projects soon. Again, a special shout out to Andrew Kane and everyone else who worked on parallel index builds.

The pgvector extension adds vector similarity search capabilities to your PostgreSQL database. To use the extension, you have to first create it in your database. You can install the extension, by connecting to your database and running the CREATE EXTENSION command from the psql command prompt: