indexify
tantivy
indexify | tantivy | |
---|---|---|
4 | 48 | |
250 | 10,017 | |
32.0% | 2.6% | |
9.9 | 9.1 | |
7 days ago | about 8 hours ago | |
Rust | Rust | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
indexify
-
How I got my first Rust job by doing open-source
Around 2 weeks ago now, someone opened an issue on OasysDB to integrate it to his platform, Indexify, an open-source platform to extract and process various unstructured data from different sources for generative AI apps in real-time.
-
I accidentally built a meme search engine
These hacks/side projects are amazing! I feel we will see a lot of creativity as tools to build data intensive AI applications become easier.
We built and open sourced Indexify https://github.com/tensorlakeai/indexify to make it easy to build resilient pipelines to combine data with many different models and transformations to build applications that relies on embedding or any other metadata extracted by models from Videos, Photos and any documents!
I didn’t know about SigClip, the author mentioned on the blog, need to add this to our library :) I also found it incredible that he generated the crawler with Claude! This is the type of boilerplate I hope we don’t have to write in the future
- Indexify -Scalable, realtime, continuous indexing engine–Unstructured Data to AI
-
Job: Rust + Retrieval Systems at Etsy
OP I sent you a message, not sure if you have seen it! I am working on an open source project in the retrieval space - https://github.com/diptanu/indexify I would love to chat more with you guys and hear what you are doing :)
tantivy
-
SeekStorm VS tantivy - a user suggested alternative
2 projects | 22 Mar 2024
-
What is Hybrid Search?
Tantivy - a full-text indexing library written in Rust. Has a great performance and featureset.
- Tantivy – Fast, OSS full-text search library in Rust
-
RAG Using Unstructured Data and Role of Knowledge Graphs
By this I presume you mean build a search index that can retrieve results based on keywords? I know certain databases use Lucene to build a keyword-based index on top of unstructured blobs of data. Another alternative is to use Tantivy (https://github.com/quickwit-oss/tantivy), a Rust version of Lucene, if building search indices via Java isn't your cup of tea :)
Both libraries offer multilingual support for keywords, I believe, so that's a benefit to vector search where multilingual embedding models are rather expensive.
-
Show HN: Quickwit – OSS Alternative to Elasticsearch, Splunk, Datadog
We also implemented our schemaless columnar storage optimized for object storage.
The inverted index and columnar storage are part of tantivy [0], which is the fastest search library out there. We maintain it and we decided to build the distributed engine on top of it.
[0] tantivy github repo: https://github.com/quickwit-oss/tantivy
-
Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
The issue for geo search is here: https://github.com/quickwit-oss/tantivy/issues/44
-
Grimoire - A recipe management application.
Search index : Custom-built using tantivy.
-
A Compressed Indexable Bitset
The roaring bitmap variant is used only for the optional index (1 docid => 0 or 1 value) in the columnar storage (DocValues), not for the inverted index. Since this is used for aggregation, some queries may be a full scan.
The inverted index in tantivy uses bitpacked values of 128 elements with a skip index on top.
> I didn't follow the rest of your comment, select is what EF is good at, every other data structure needs a lot more scanning once you land on the right chunk. With BMI2 you can also use the PDEP instruction to accelerate the final select on a 64-bit block
The select for the sparse codec is a [simple array index access](https://github.com/quickwit-oss/tantivy/blob/main/columnar/s...), that is hard to beat. Compression is not good near the 5k threshold though.
-
Job: Rust + Retrieval Systems at Etsy
Hi /r/rust, I’m a SWE on Etsy’s Retrieval Systems team where we’re building a platform based on rust and tantivy (https://github.com/quickwit-oss/tantivy). We’re looking to bring two new engineers onto the team.
-
Announcing Velo - Your Rust-Powered Brainstorming and Note-Taking Tool
Quick Search: Easily find specific notes with Velo's fuzzy-search feature, powered by tantivy. tantivy might have been a little overkill, but it was really easy to integrate.
What are some alternatives?
floneum - A toolkit for controllable, private AI on consumer hardware in rust
sonic - 🦔 Fast, lightweight & schema-less search backend. An alternative to Elasticsearch that runs on a few MBs of RAM.
text-embeddings-inference - A blazing fast inference solution for text embeddings models
surrealdb - A scalable, distributed, collaborative, document-graph database, for the realtime web
searchGPT - Grounded search engine (i.e. with source reference) based on LLM / ChatGPT / OpenAI API. It supports web search, file content search etc.
milli - Search engine library for Meilisearch ⚡️
burn - Burn is a new comprehensive dynamic Deep Learning Framework built using Rust with extreme flexibility, compute efficiency and portability as its primary goals.
MeiliSearch - A lightning-fast search API that fits effortlessly into your apps, websites, and workflow
Queryable - Run OpenAI's CLIP model on iOS to search photos.
quickwit - Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
smartgpt - A program that provides LLMs with the ability to complete complex tasks using plugins.
fselect - Find files with SQL-like queries