Our great sponsors
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
salsa
A generic framework for on-demand, incrementalized computation. Inspired by adapton, glimmer, and rustc's query system.
-
No, at most we would've invested more in e.g. ensuring¹ you cannot accidentally (or intentionally) mix-and-match shared objects not built by the same build process, or in ways to reduce the cost² of safe ("full relro") dynamic linking.
¹ via forcing the dynamic linker to eagerly resolve some symbols, global constructors, etc. - see https://github.com/rust-lang/rust/issues/73917#issuecomment-... for some recent discussion
² such as exporting a single base symbol per dylib (with a hash of code/data contents in the symbol name, for integrity), and statically resolving imports (to constant offsets from that base symbol), when linking object files into dylibs/executables - because of how position-independent code indirects through the "GOT", in practice this would mean that the dynamic linker would only need to lookup one symbol per dependency .so instead of one per import (of which there can be hundreds or thousands)
Also, "dynamic linking" in Rust was never really "dynamic" or about "drop-in replacement to avoid rebuilds", it was only about the storage reuse of "shared objects", with the "late-binding" semantics inherent in dynamic linking a negative, not a positive.
For example, rustc, rustdoc, clippy, miri, etc. are all relatively small executables that link against librustc_driver-.so - on a recent nightly that's 120MiB, with only static linking that'd be half a GiB of executables instead. The relationship is "one .so shared between N executables" not* "one .so per library". Also, if we could remove runtime symbol lookup but keep the separate .so (like ² above), we totally would.
---
At the same time, Rust's continued evolution depends on being able to (almost) "never ask for permission" when changing internals that were never guaranteed to be one thing or another.
Rust recently fixed most platforms to use simpler and more performant locks based on futex(-like) APIs (see https://github.com/rust-lang/rust/issues/93740 and PRs like https://github.com/rust-lang/rust/pull/95035), and this meant that e.g. Mutex has a 32-bit (atomic) integer where a pointer used to be.
An even more aggressive change was decoupling Ipv{4,6}Addr/SocketAddrV{4,6} from the C types they used to wrap internally (https://github.com/rust-lang/rust/pull/78802) - that one actually required library authors to fix their code in a few cases (that were very incorrectly reinterpreting the std types as the C ones, without this ever being officially supported or condoned).
Imagine trying to do that in C++. The stories I keep hearing from people involved in the C++ standard are tragic, more and more otherwise-uncontroversial improvements are being held back by poorly-motivated stubbornness around ABI stability. Large-scale users of C++, like Google, have long since migrated from C++'s standard library to their own replacements (e.g. abseil) for many of their codebases, and I kept hearing Google were "forking C++" (over the frozen ABI debacle specifically), long before it came out as "Carbon".
---
The technology to coherently³ avoid today's rebuild-the-world cost in compiled languages⁴ isn't here yet IMO.
³ as in, guarantee identical behavior to the full rebuild from scratch, without introducing inconsistencies that could cause logic errors, memory corruption, etc.
⁴ some/most C libraries of course being the biggest exception - and while it's possible to intentionally design libraries like this in any language that can do FFI, it's fundamentally antithetical to "zero-cost abstractions" and compiler optimizations - nowadays the latter are forced even on otherwise-patchable C dependencies via LTO, in the interest of performance
What I'm referring to is incremental recompilation with the following properties:
So there are only a few examples of existing systems being re-written for Optane.
There was Redis [1] which was around ~10% slower than the memory only version but gained you durability. And there was an implementation of Spark Shuffle [2] which was 1.3x - 3.2x faster but that isn't really stressing I/O as much as other use cases.
For a filesystem you can store the entire metadata in Optane so EXISTS/LIST type operations would see the full benefit e.g. order of magnitude better than NVME.
[1] https://github.com/pmem/pmem-redis
[2] https://databricks.com/session_na20/accelerating-apache-spar...
- the practical approach to this is to split workload into work units (aka tasks/queries/etc.) and then force information flow through centralized "request"/"query" APIs that automatically track dependencies - see https://github.com/salsa-rs/salsa for more information
- outside of a compiler, the only examples I'm aware of are build systems like tup (https://gittup.org/tup/) or RIKER (https://github.com/curtsinger-lab/riker) which use filesystem sandboxing (FUSE for tup, ptrace/seccomp-BPF for RIKER) for "automatically correct" build dependencies
- outside of a compiler, the only examples I'm aware of are build systems like tup (https://gittup.org/tup/) or RIKER (https://github.com/curtsinger-lab/riker) which use filesystem sandboxing (FUSE for tup, ptrace/seccomp-BPF for RIKER) for "automatically correct" build dependencies