polars VS faiss

https://github.com/pola-rs/polars/releases/tag/py-0.19.0

- handling of categoricals in polars seemed a little underbaked, though my main complaint, that categories cannot be pre-defined, seems to have been recently addressed: https://github.com/pola-rs/polars/issues/10705

That turned out to be related to pola-rs/polars#11912, and this linked comment provided a deceptively simple solution - use PARSE_DECLTYPES when creating the connection:

Looks like somebody requested it after reading your TIL. https://github.com/pola-rs/polars/issues/12493#issuecomment-...

It will be in the next release. (later today?)

I am a maintainer for a dataframe interface called polars

For example with polars, the python library extensively uses named arguments, but in rust we have to use either a builder pattern or macros. The builder pattern tends to be much more verbose than the named argument equivalent. There is currently a draft PR implementing python style named arguments for some of the most common functions.

Seeing a lot of great libraries coming out with python bindings in the data world e.g delta-rs Polars. I see it growing in this space as a C++ alternative

Or just use FAISS https://github.com/facebookresearch/faiss

You can try txtai (https://github.com/neuml/txtai) with a Faiss backend.

This Faiss wiki article might help (https://github.com/facebookresearch/faiss/wiki/Indexing-1G-v...).

For example, a partial Faiss configuration with 4-bit PQ quantization and only using 5% of the data to train an IVF index is shown below.

faiss={"components": "IVF,PQ384x4fs", "sample": 0.05}

If you want to experiment with vector stores, you can do that locally with something like faiss which has good platform support: https://github.com/facebookresearch/faiss

Doing full retrieval-augmented generation (RAG) and getting LLMs to interpret the results has more steps but you get a lot of flexibility, and there's no standard best-practice. When you use a vector DB you get the most similar texts back (or an index integer in the case of faiss), you then feed those to an LLM like a normal prompt.

The codifer for the RAG workflow is LangChain, but their demo is substantially more complex and harder-to-use than even a homegrown implementation: https://news.ycombinator.com/item?id=36725982

According to this documentation page, faiss-gpu is only supported on Linux, not on Windows.

As the size of the codebase grows, storing and searching through embeddings in memory becomes inefficient. This is where vector databases come into play. Tools like Milvus, Faiss, and others are designed to handle large-scale vector data and provide efficient similarity search capabilities. I've wrtten about how to also use sqlite to store vector embeddings. By integrating a vector database, you can scale your code search tool to handle much larger codebases without compromising on search speed.

But FAISS has their own version ("FastScan") https://github.com/facebookresearch/faiss/wiki/Fast-accumula...

https://github.com/facebookresearch/faiss

If you want to go larger you could still use some simple setup in conjunction with faiss, annoy or hnsw.

Facebook research for having open source licensed an amazing vector based indexing library