indexify VS tantivy

2. PDF Extraction -> PDF -> Markdown -> Chunking -> Embedding, NER (https://github.com/tensorlakeai/indexify/blob/main/docs/docs...)

Application Layer - Indexify works as a retriever in the LLM application stack, so you can use it pretty easily with your existing applications. Call the retriever API over HTTP to get extracted data from Indexify, and that's pretty much all the integration you need to search or retrieve data.

You could use composable extractors and chain them together to build complex real time data pipelines that work with any unstructured data.

Since this is HN, I have the liberty to talk some technical details :)

How is it Real Time?

Around 2 weeks ago now, someone opened an issue on OasysDB to integrate it to his platform, Indexify, an open-source platform to extract and process various unstructured data from different sources for generative AI apps in real-time.

These hacks/side projects are amazing! I feel we will see a lot of creativity as tools to build data intensive AI applications become easier.

We built and open sourced Indexify https://github.com/tensorlakeai/indexify to make it easy to build resilient pipelines to combine data with many different models and transformations to build applications that relies on embedding or any other metadata extracted by models from Videos, Photos and any documents!

I didn’t know about SigClip, the author mentioned on the blog, need to add this to our library :) I also found it incredible that he generated the crawler with Claude! This is the type of boilerplate I hope we don’t have to write in the future

OP I sent you a message, not sure if you have seen it! I am working on an open source project in the retrieval space - https://github.com/diptanu/indexify I would love to chat more with you guys and hear what you are doing :)

Fun fact: We've implemented binary embedding search [1] without the need for a specialized vector database. Instead, we create dimensional tokens like 'embedding_0_0', 'embedding_1_0', and so on, and we harness the robust capabilities of Tantivy [2]. We're really satisfied with the exceptional quality and performance this approach yields. Moreover, Tabby remains a single binary, integrating all these components seamlessly.

[1] My binary vector search is better than your FP32 vectors: https://blog.pgvecto.rs/my-binary-vector-search-is-better-th...

[2] Tantivy: https://github.com/quickwit-oss/tantivy

| Hm, I am interested, but I would love to use it as a rust lib and just have rust types instead of some json config...

Yes that's how you use tantivy normally, not sure which json config you mean.

`tantivy-cli` is more like a showcase, https://github.com/quickwit-oss/tantivy is the actual project.

Tantivy - a full-text indexing library written in Rust. Has a great performance and featureset.

By this I presume you mean build a search index that can retrieve results based on keywords? I know certain databases use Lucene to build a keyword-based index on top of unstructured blobs of data. Another alternative is to use Tantivy (https://github.com/quickwit-oss/tantivy), a Rust version of Lucene, if building search indices via Java isn't your cup of tea :)

Both libraries offer multilingual support for keywords, I believe, so that's a benefit to vector search where multilingual embedding models are rather expensive.

We also implemented our schemaless columnar storage optimized for object storage.

The inverted index and columnar storage are part of tantivy [0], which is the fastest search library out there. We maintain it and we decided to build the distributed engine on top of it.

[0] tantivy github repo: https://github.com/quickwit-oss/tantivy

The issue for geo search is here: https://github.com/quickwit-oss/tantivy/issues/44

Search index : Custom-built using tantivy.

The roaring bitmap variant is used only for the optional index (1 docid => 0 or 1 value) in the columnar storage (DocValues), not for the inverted index. Since this is used for aggregation, some queries may be a full scan.

The inverted index in tantivy uses bitpacked values of 128 elements with a skip index on top.

> I didn't follow the rest of your comment, select is what EF is good at, every other data structure needs a lot more scanning once you land on the right chunk. With BMI2 you can also use the PDEP instruction to accelerate the final select on a 64-bit block

The select for the sparse codec is a [simple array index access](https://github.com/quickwit-oss/tantivy/blob/main/columnar/s...), that is hard to beat. Compression is not good near the 5k threshold though.