Our great sponsors
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
Two years ago I started working on an Erlang inspired language, that evolved into lunatic. Lunatic nowadays allows you to take "any" language compiled to WebAssembly and expose mechanisms of lightweight processes to it. Originally it started as a lua variant (that's where the name comes from), but today we have the best support for rust.
Taking onto such a big project can be scary and overwhelming, so I like to "cheat" a bit. Instead of developing a M:N scheduler I picked an already mature and proven one from the Rust ecosystem: tokio. Then I just needed to develop a virtual stacks solution that works well with the scheduler. Instead of inventing my own byte-code I just picked WebAssembly, it's just a small abstraction above machine code and has mature JIT compiler libraries that generate code close to native speed. Then again, I just needed to figure out how to do reduction counting and insert preemption points into WebAssembly code during loading.
For concurrency/parallelism, you launch at most 2 * CPU Cores, PIN them and use a fast broker to spread the task (like a ring buffer or an MPSC). But you keep linear scan, tight loops, SIMD friendly data, on each. You are not switching context that much, and instead, bet you will process the batch fast. (CPUs are fast today!)
One example that might help you get started: https://github.com/darius/cant/tree/master/examples/squirm
The Beam Languages repo is filled with projects to build on top of Erlang and the BEAM to take inspiration from.
Related posts
- Hyperbridge: Fast multi-producer, multi-consumer unbounded channel in Rust
- Loole, A safe sync/async multi-producer, multi-consumer channel, Boosted Async Performance Up to 17%
- appreciating fearless concurrency
- Rust has a small standard library (and that's ok)
- Request-response communication between threads?