cosmicverge VS flume

I have a pretty-well-tested codebase that I'm almost ready to integrate into Cosmic Verge. While I have plenty of work remaining on PliantDB, I'm excited at the prospect of replacing PostgreSQL and Redis in Cosmic Verge potentially next month.

I also like flume, it has impressive performance (although not the best). More importantly, it's written only with safe rust. https://github.com/zesterer/flume

The most commonly suggested replacement for mspc is crossbeam-channel; flume is also relatively popular.

It's not officially deprecated, but the alternatives on crates.io are considered better. flume and crossbeam-channel feature less unsafe code and offer better performance. Benchmarks.

it's slow (checkout flume's benchmarks for example)

For concurrency/parallelism, you launch at most 2 * CPU Cores, PIN them and use a fast broker to spread the task (like a ring buffer or an MPSC). But you keep linear scan, tight loops, SIMD friendly data, on each. You are not switching context that much, and instead, bet you will process the batch fast. (CPUs are fast today!)

First, I was becoming more and more confident that the channel library Daxpedda and I fell in love with, flume, was misbehaving, but I couldn't seem to reproduce it outside of the massive PliantDB codebase. I finally called up Daxpedda on Discord and screen shared my debugging session, showing him how the tests succeeded if I retained a channel. If I allowed the sender to drop after successfully sending, sometimes the tests would fail. He agreed, something was odd. It took me a while, but I finally whittled it down to about 30 lines of code and reported the issue. In an amazingly quick fashion, the maintainer fixed the issue and released an update. And for the record, I still fully love and recommend this library if you're mixing async and non-async code using channels. It's a wonderful implementation.

Sort of? Here's the closest thing you'll probably find: https://crates.io/crates/flume

My first try was with flume, which uses a single producer, multi-consumer setup, and while it compiles and runs, it's magnitudes slower than the serial version. Here's a psuedocode rendition: