imdb-rename VS wg-allocators

I wrote a tool that did something related a while back using IMDb data: https://github.com/BurntSushi/imdb-rename

This might be of interest: https://github.com/BurntSushi/imdb-rename

What a shit take. The article itself is perhaps a nice light overview of 101-ish level concepts, although knowing how and when to apply them in a real engineering context is not something I would consider 101 level. And certainly, building something that is actually at the scale of GitHub Search is nowhere near 101 level.

This is what a 101-level inverted index implementation looks like: https://github.com/BurntSushi/imdb-rename

In other words, absolutely nothing like what GitHub built. Nowhere close.

imdb-rename is an example of a tool that memory maps FSTs on disk in order to execute fulltext searches very quickly on the command line.

> How else would you lazy-load a database of (say) 32GB into memory, almost instantly?

That's what the fst crate[1] does. It's likely working at a lower level of abstraction than you intend. But the point is that it works, is portable and doesn't require any cooperation from the OS other than the ability to memory map files. My imdb-rename tool[2] uses this technique to build an on-disk database for instantaneous searching. And then there is the regex-automata crate[3] that permits deserializing a regex instantaneously from any kind of slice of bytes.[4]

I think you should maybe provide some examples of what you're suggesting to make it more concrete.

[1] - https://crates.io/crates/fst

[2] - https://github.com/BurntSushi/imdb-rename

[3] - https://crates.io/crates/regex-automata

[4] - https://docs.rs/regex-automata/0.1.9/regex_automata/#example...

Tracking issue for Storages, and a TLDR on what it is

I must have gotten confused, since from your brief discussion with CAD97 it seemed like there was a way for the concepts to live separately and that Storage could complicate things in comparison. But if implementing Allocator in terms of Storage is basically equivalent and Storage is flexible enough that I could write one to pass memory out to unsafe code, that works just as well.

https://rust-lang.github.io/rfcs/1974-global-allocators.html was the original RFC.

My vague understanding is that there's a working group https://github.com/rust-lang/wg-allocators

The further I get from working on Rust day to day, the less I know about these things, so that's all I've got for you.

If you self-reference using pointers and guarantee the struct will never move, you don't even need unsafe. If you self-reference using offsets from the struct's base pointer, you need a splash of unsafe but your struct can be freely moved without invalidating its self-referential "pointers".

Per-struct allocators are a work in progress (see https://github.com/rust-lang/wg-allocators/issues/48).

Not sure what "non thread local addresses" means, but in my experience Rust is pretty good at sending data between threads (without moving it).

Besides that allocation is not really a problem for no_std. It's resolved by using alloc crate directly, so anything usable with custom allocators is supported. Example in dasp sources - https://github.com/RustAudio/dasp/blob/master/dasp_slice/src/boxed.rs#L14-L19 . Also worth looking at this issue to check what is usable already - https://github.com/rust-lang/wg-allocators/issues/7

But that's on track for rust as well: https://github.com/rust-lang/wg-allocators/issues/7

Here: https://github.com/rust-lang/rfcs/pull/1398. there is also a working group for this: https://github.com/rust-lang/wg-allocators.

It's unstable. wg-allocators contains discussions about design and a tracking issue for collections that need an allocator https://github.com/rust-lang/wg-allocators/issues/7