90x Faster Than Pgvector – Lantern's HNSW Index Creation Time

• lantern

  5 646 9.6 C

  PostgreSQL vector database extension for building AI applications

  

This extension is licensed under the Business Source License[0], which makes it incompatible with most DBaaS offerings. The BSL is a closed-source license. Good choice for Lantern, but unusable for everyone else.

Some Postgres offerings allow you to bring your own extensions, for instance Neon[1], where I work. I tried to look at AWS docs for you, but couldn't find anything about that. I did find Trusted Language Extensions[2], but that seems to be more about writing your own extension. Couldn't find a way to upload arbitrary extensions.

[0]: https://github.com/lanterndata/lantern/commit/dda7f064ca80af...

• marqo

  114 4,111 9.3 Python

  Unified embedding generation and search engine. Also available on cloud - cloud.marqo.ai

  

That sounds much longer than it should. I am not sure on your exact use-case but I would encourage you to check out Marqo (https://github.com/marqo-ai/marqo - disclaimer, I am a co-founder). All inference and orchestration is included (no api calls) and many open-source or fine-tuned models can be used.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• jvector

  3 1,257 9.7 Java

  JVector: the most advanced embedded vector search engine

Nice to see people care about index construction time.

JVector scales linearly to at least 32 cores and may be the only graph-based vector index designed around nonblocking data structures (as opposed to fine-grained locks): https://github.com/jbellis/jvector/

JVector indexes the Sift1M dataset in under 19s on a 32 core aws box (m6i.16xl), compared to 50s for Lantern in the article.

• usearch

  20 1,629 9.8 C++

  Fast Open-Source Search & Clustering engine × for Vectors & 🔜 Strings × in C++, C, Python, JavaScript, Rust, Java, Objective-C, Swift, C#, GoLang, and Wolfram 🔍

  

• pgrx

  13 3,235 9.5 Rust

  Build Postgres Extensions with Rust!

  

(disclosure, i work at supabase and have been developing TLEs with the RDS team)

Trusted Language Extensions refer to an extension written in any trusted language. In this case Rust, but it also includes: plpgsql, plv8, etc. See [0]

> PL/Rust is a more performant and more feature-rich alternative to PL/pgSQL

This is only partially true. plpgsql has bindings to low-level Postgres APIs, so in some cases it is just as fast (or faster) than Rust.

> Building a vector index (or any index for that matter) inside Postgres is a more involved process and can not be done via the UDF interface, be it Rust, C or PL/pgSQL

Most PG Rust extensions are written with the excellent pgrx framework [1]. While it doesn't have index bindings right now, I can certainly imagine a future where this is possible[2].

All that said - I think there are a lot of hoops to jump through right now and I doubt it's worth it for the Latern team. I think they are right to focus on developing a separate C extension

[0] TLE: https://supabase.com/blog/pg-tle

[1] pgrx: https://github.com/pgcentralfoundation/pgrx

[2] https://github.com/pgcentralfoundation/pgrx/issues/190#issue...

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project