tantivy VS Iron

Tantivy - a full-text indexing library written in Rust. Has a great performance and featureset.

By this I presume you mean build a search index that can retrieve results based on keywords? I know certain databases use Lucene to build a keyword-based index on top of unstructured blobs of data. Another alternative is to use Tantivy (https://github.com/quickwit-oss/tantivy), a Rust version of Lucene, if building search indices via Java isn't your cup of tea :)

Both libraries offer multilingual support for keywords, I believe, so that's a benefit to vector search where multilingual embedding models are rather expensive.

We also implemented our schemaless columnar storage optimized for object storage.

The inverted index and columnar storage are part of tantivy [0], which is the fastest search library out there. We maintain it and we decided to build the distributed engine on top of it.

[0] tantivy github repo: https://github.com/quickwit-oss/tantivy

The issue for geo search is here: https://github.com/quickwit-oss/tantivy/issues/44

Search index : Custom-built using tantivy.

The roaring bitmap variant is used only for the optional index (1 docid => 0 or 1 value) in the columnar storage (DocValues), not for the inverted index. Since this is used for aggregation, some queries may be a full scan.

The inverted index in tantivy uses bitpacked values of 128 elements with a skip index on top.

> I didn't follow the rest of your comment, select is what EF is good at, every other data structure needs a lot more scanning once you land on the right chunk. With BMI2 you can also use the PDEP instruction to accelerate the final select on a 64-bit block

The select for the sparse codec is a [simple array index access](https://github.com/quickwit-oss/tantivy/blob/main/columnar/s...), that is hard to beat. Compression is not good near the 5k threshold though.

Hi /r/rust, I’m a SWE on Etsy’s Retrieval Systems team where we’re building a platform based on rust and tantivy (https://github.com/quickwit-oss/tantivy). We’re looking to bring two new engineers onto the team.

Quick Search: Easily find specific notes with Velo's fuzzy-search feature, powered by tantivy. tantivy might have been a little overkill, but it was really easy to integrate.

I've written many things using Iron and it's been fine. It's not particularly developed any more but I am not aware of any major outstanding issues.

I don't miss the time when we basically only had Iron

iron

There're many web framework in rust, such as SergioBenitez/Rocket , actix/actix-web ,poem-web/poem , iron/iron . How do I chiose, anyone suggestion？

I'm not sure which definition of backend you are thinking of here. In case you simply mean "server side", here are a couple of libraries that might be of use: iron - been a while since I used it, used it for a couple smaller projects rocket (nightly only) - no personal experience, but a lot of people seem to like it diesel - a bit complex to wrap your head around, but once you get the idea it's really nice. Definitely check out the examples.

it's taken by a web framework https://github.com/iron/iron

Your comment touches on a few misconceptions I see a lot.

Firstly, `reqwest` exposes both an async and a synchronous API, allowing the developer to choose which one to use. They are largely interchangeable code-wise. [1]

Secondarily, and more broadly, async is possible to opt out of. You must understand that most web and network related libraries will be async by default for performance, because people who write in Rust and people who write web servers typically care greatly about performance. This is the intersection of those two groups. That being said, there are options outside of that ecosystem. [2]

If you truly want to use an asynchronous library without migrating your application to run entirely on an async runtime like tokio, you can run it inside of a synchronous function without much trouble. I've put together a playground link for you. [3]

1. https://docs.rs/reqwest/0.11.2/reqwest/blocking/index.html

2. Iron: https://github.com/iron/iron

There are several web frameworks for Rust: Rocket, Actix, Warp, Iron - but only Actix has released a stable 1.0 release, and there has been considerable controversy over how it uses unsafe Rust.