The Best Way to Use Text Embeddings Portably Is with Parquet and Polars

1. lancedb

   1 14 11,142 9.8 Rust

   Developer-friendly OSS embedded retrieval library for multimodal AI. Search More; Manage Less.

   

   For another library that has great performance and features like full text indexing and the ability to version changes I’d recommend lancedb https://lancedb.github.io/lancedb/

   Yes, it’s a vector database and has more complexity. But you can use it without creating indexes and it has excellent polars and pandas zero copy arrow support also.

2. SaaSHub

   www.saashub.com sponsored

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. mtg-embeddings

   2 2 62 8.0 Jupyter Notebook

   Code used to create text embeddings of all Magic: The Gathering cards.

   

   In general I like to send structured data (see the input format here: https://github.com/minimaxir/mtg-embeddings), but the ModernBERT base for the embedding model used here specifically has better benefits for structured data compared to previous models. That's worth another blog post explaining why.

4. awesome-polars

   3 26 1,116 8.3

   A curated list of Polars talks, tools, examples & articles. Contributions welcome !

   

5. portable-hnsw

   4 5 111 0.0 HTML

   What if an HNSW index was just a file, and you could serve it from a CDN, and search it directly in the browser?

   

   I thought about this some more and did some research - and found an indexing approach using HNSW, serialized to parquet, and queried from the browser here:

   https://github.com/jasonjmcghee/portable-hnsw

   Opens up efficient query patterns for larger datasets for RAG projects where you may not have the resources to run an expensive vector database

Suggest a related project