fst VS smartstring

I'd point you towards BurntSushi's fst crate: https://github.com/BurntSushi/fst

The fst crate effectively relies on mmap for it to work right. The folks here suggesting you just use the heap might be right, but only if using the heap is actually plausible. If your dictionary is GBs big (an FST might be bigger than available memory), then copying it the heap first would be disastrous.

You'll love what we're working on if you're interested in the implementation of:- Tantivy- Meilisearch- Finite State Transducers

Here's an example of longest prefix matching using a FST which I based my approach on: https://github.com/BurntSushi/fst/pull/104/files

Finite State Transducers

As a result, I have a plan to implement a desktop version from then and I finished today with a beta version. The desktop is based on tauri, and the dictionary index algorithm is FST (it is an awesome index algorithm).

WordBueno’s data is currently derived from Wiktionary. The backend is using Rust’s warp with fst for indexing.

No you don't. I've written multiple programs that load things instantly off the file system via memory maps. See the fst crate[1], for example, which is designed to work with memory maps.

Rust "works badly with memory mapped files" doesn't mean, "Rust can't use memory mapped files." It means, "it is difficult to reconcile Rust's safety story with memory maps." ripgrep for example uses memory maps because they are faster sometimes, and its safety contract[2] is a bit strained. But it works.

[1] - https://github.com/BurntSushi/fst/

[2] - https://docs.rs/grep-searcher/0.1.7/grep_searcher/struct.Mma...

Your use case sounds like it will involve a lot of small strings that use a subset of UTF-8. If you’re concerned about performance, you could look into something like smartstring. Sixbit also looks interesting, but it looks like it won’t give you any more characters and it’d probably require additional computation to do the conversion (and they’d have to be converted back out).

> If you have a long-running async function, then pass parameters by value! If you have a polymorphic async function, then return your result in a Box.

I've taken to making heavy use of the smallvec and smartstring crates for this. Most lists and strings are small in practice. Using smallvec / smartstring lets you keep most clone() calls allocation-free. This in turn lets you use owned objects, which are easier to reason about - for you and the borrow checker. And you keep a lot of the performance of just passing around references.

I tried to use async rust a couple of years ago, and fell on my face in the process. Most of my rust at the moment is designed to compile to wasm - and then I'm leaning on nodejs for networking and IO. Writing async networked code is oh so much easier to reason about in javascript. When GAT, TAIT and some other language features to fix async land I'll muster up the courage to make another attempt. But rust's progress at fixing these problems feels painfully slow.

https://crates.io/crates/smallvec / https://crates.io/crates/smartstring

Just to point out, smartstring no longer assumes String memory layout. From the changelog:

There's some really good crates that implement SSO floating around - eg, SmartString. But I agree - its a pity they're needed. Swift built this into the core string type in the language. I think that was the right call.

Comparatively: * SmolStr can inline up to 22 bytes but does not adjust down for 32-bit architectures, meaning it's potentially wasting memory on 32-bit archs. Similarly though it's immutable and Clone is O(1) * SmartString can inline up to 23 bytes, but it's mutable and Clone is O(n). Also this crate makes assumptions about the memory layout of a String, which in theory should be fine, but is a slight caveat.

SmartString is used to store identifiers (which tends to be short, fewer than 23 characters, and ASCII-based) because they can usually be stored inline. Map keys now also use SmartString.

I’ve been using smartstrings, which is both excellent and maintained. https://github.com/bodil/smartstring