hnswlib VS postgres

How does this compare to hsnwlib - is it faster? https://github.com/nmslib/hnswlib

Last week I released Moodflix (https://moodflix.streamlit.app), a movie recommendation engine based to find movies based on your mood.

Moodflix was created on top of a movie dataset of 10k movies from The Movie Database. I vectorised the films using Hugging Face's T5 model (https://huggingface.co/docs/transformers/model_doc/t5) using the film's plot synopsis, genres and languages. Then I indexed the vectors using hnswlib (https://github.com/nmslib/hnswlib). LLMs can understand a movie's plot pretty well and distill the similarities between a user's query (mood) to the movie's plot and genres.

I have got feedback from close friends around linking movies to other review sites like IMDB or Rotten Tomatoes, linking movies to sites to stream the movie and adding movie posters. I would also love to hear from the community what things you like, what you want to see and what things you consider can be improved.

Actually the "ef" is not epsilon. It is a parameter of the HNSW index: https://github.com/nmslib/hnswlib/blob/master/ALGO_PARAMS.md...

If you want to go larger you could still use some simple setup in conjunction with faiss, annoy or hnsw.

Many vector databases are using Hnswlib and that is a supported vector index alongside Faiss and Annoy.

hnswlib (https://github.com/nmslib/hnswlib) is a strong alternative to faiss that I have enjoyed using for multiple projects. It is simple and has great performance on CPU.

After working through several projects that utilized local hnswlib and different databases for text and vector persistence, I integrated hnswlib with sqlite to create an embedded vector search engine that can easily scale up to millions of embeddings. For self-hosted situations of under 10M embeddings and less than insane throughput I think this combo is hard to beat.

https://github.com/jiggy-ai/hnsqlite

https://github.com/nmslib/hnswlib

Used it to index 40M text snippets in the legal domain. Allows incremental adding.

I love how it just works. You know, doesn’t ANNOY me or makes a FAISS. ;-)

3000 texts doesn't sound like to many, so may be a brute force cos calculation to find the most similar vector would work. If that's taking too much time, may be look at KNN or ANN modules to speed up finding the most similar vector. I use hsnwlib in knn mode for this. SOrt through about 350,000 vectors in about 30-50 msec.

hnswlib is in cpp and has python bindings (you should be able to make your own for other languages).

https://github.com/nmslib/hnswlib

embeddings - The embeddings index file. This is an Approximate Nearest Neighbor (ANN) index with either Faiss (default), Hnswlib or Annoy, depending on the settings.

- Now, the connection between our API servers and the database was slow (a few hundred ms per query), so we moved to self-hosting Postgres which was pretty painful. We tried to use https://github.com/supabase/postgres, but the documentation was very lacking and we had to make a bunch of modifications to get it to work. After we got it working, it was pretty smooth though - pretty easy to implement backups, etc.

I think Supabase generally does good work, but I don't think they can be given credit for pgvector, if that's what you're indicating (I might have misread).

As I understand, Andrew Kane is the principal author of pgvector, and has worked on it for almost two years before Supabase added support for it.

See also https://github.com/pgvector/pgvector/issues/54 and https://github.com/supabase/postgres/pull/472.

we merged the pgvector PR about 2 weeks ago (https://github.com/supabase/postgres/pull/472). If you're missing anything for your CLI don't hesitate to reach out and we'll see if we can integrate it into the product (my email is in my profile)

as an aside, Pinecone looks great

One of the things they recommend is separating the PostgreSQL DB from the rest of the stack. And they also provide an ansible playbook to set up a postgres DB: https://github.com/supabase/postgres/blob/625899e687047a9da658f3f8cc6dd91ac9769694/ansible/playbook.yml

> entire solution from docker

We PR'd this into our docker-compose today [0]. We're always a bit slammed during Launch Week, so if you spot any problems let use know and we'll patch it up asap.

The extension is also deployed directly into our PG bundle [1] which is available in docker [2]

> The Gui for adding roles and tying them to postgres access is very slick with hasura. Is this done manually via SQL commands with supabase?

I haven't tried Hasura so I don't know if this is a direct comparison. pg_graphql works with Postgres Row Level Security - we provide a GUI for this in our Dashboard, but they are also just native PG Policies, so you can write them in raw SQL

[0] https://github.com/supabase/supabase/pull/6138/files#diff-41...

[1] https://github.com/supabase/postgres

[2] https://hub.docker.com/r/supabase/postgres

and can also PR if it's something useful for everyone: https://github.com/supabase/postgres

Show HN: Postgres Docker image with common extensions\ (23 comments)

Hi! I'm one of the contributors to the repo. Just to clarify, our Docker image [0] only contains the latest version of Postgres (13) and the common extensions listed out here [1]. All the other features such as this [2] and this [3] are only available in the AWS EC2 or DO droplet images. We've since updated our README to make that clearer :-)

You can still connect the DB with a PgBouncer image spun up in another container however. Unfortunately, I can't really recommend you which one since there doesn't seem to be an official Docker image for PgBouncer and I myself have never tried any of the existing ones out there. If you're looking to use PostgREST however, they do have an official Docker image that you can use over here [4].

[0]: https://hub.docker.com/r/supabase/postgres

[1]: https://github.com/supabase/postgres#extensions

[2]: https://github.com/supabase/postgres#enhanced-security

[3]: https://github.com/supabase/postgres#additional-goodies

[4]: https://hub.docker.com/r/postgrest/postgrest/