direct-io VS wg-allocators

I wrote this for Node.js, which is a native binding in C, exposing cross platform functionality: https://github.com/ronomon/direct-io

Although if it's a new project and you're used to C, I would recommend also taking a good look at Zig (https://ziglang.org/), because it's so explicit about alignment compared to C, and makes alignment a first-class part of the type system, see this other comment of mine that goes into more detail: https://news.ycombinator.com/item?id=25801542

Something that will also help, is setting your minimum IO unit to 4096 bytes, the Advanced Format sector size, because then your Direct IO system will just work, regardless of whether sysadmins swap disks of different sector sizes from underneath you. For example, a minimum sector size of 4096 bytes will work not only for newer AF disks but also for any 512 byte sector disks.

Lastly, Direct IO is actually more a property of the file system, not necessarily the OS (e.g. Linux), so you will some file systems on Linux that return EINVAL when you try to open a file descriptor with O_DIRECT, simply because they don't support O_DIRECT (e.g. a macOS volume accessed from within a Linux VM) so that should be your way of testing for support, not only the OS.

Tracking issue for Storages, and a TLDR on what it is

I must have gotten confused, since from your brief discussion with CAD97 it seemed like there was a way for the concepts to live separately and that Storage could complicate things in comparison. But if implementing Allocator in terms of Storage is basically equivalent and Storage is flexible enough that I could write one to pass memory out to unsafe code, that works just as well.

https://rust-lang.github.io/rfcs/1974-global-allocators.html was the original RFC.

My vague understanding is that there's a working group https://github.com/rust-lang/wg-allocators

The further I get from working on Rust day to day, the less I know about these things, so that's all I've got for you.

If you self-reference using pointers and guarantee the struct will never move, you don't even need unsafe. If you self-reference using offsets from the struct's base pointer, you need a splash of unsafe but your struct can be freely moved without invalidating its self-referential "pointers".

Per-struct allocators are a work in progress (see https://github.com/rust-lang/wg-allocators/issues/48).

Not sure what "non thread local addresses" means, but in my experience Rust is pretty good at sending data between threads (without moving it).

Besides that allocation is not really a problem for no_std. It's resolved by using alloc crate directly, so anything usable with custom allocators is supported. Example in dasp sources - https://github.com/RustAudio/dasp/blob/master/dasp_slice/src/boxed.rs#L14-L19 . Also worth looking at this issue to check what is usable already - https://github.com/rust-lang/wg-allocators/issues/7

But that's on track for rust as well: https://github.com/rust-lang/wg-allocators/issues/7

Here: https://github.com/rust-lang/rfcs/pull/1398. there is also a working group for this: https://github.com/rust-lang/wg-allocators.

It's unstable. wg-allocators contains discussions about design and a tracking issue for collections that need an allocator https://github.com/rust-lang/wg-allocators/issues/7