bonsaidb VS flume

I do have ideas in the issue tracker on some of the next steps towards an actual migration system.

What about https://github.com/khonsulabs/bonsaidb? Progress seems stall since last summer but very cool project

There is Nebari which is the KV part of BonsaiDB I've used both successfully (and that is currently in production)

In BonsaiDb, I define entire test suites as macros. This crate has a common trait that has multiple implementations in different crates. Each implementation needs to be tested thoroughly. For cargo test to be able to work in each crate independently, I needed to have the #[test]-annotated functions in the crate being built. By using a macro, I can define the functions in one location and invoke the macro in each crate to import the test suite into that crate.

hey, just letting you know that there already is a project called bonsai-db and some people might confuse bonsai-bt as part of that project

I'm finishing up a large refactor of BonsaiDb which will add support for using BonsaiDb in non-async code.

I'm working on a major refactoring of BonsaiDb, aiming to improve the design of several interrelated features. While it started by aiming to enable a non-async interface for BonsaiDb, I realized mid-refactor that another major refactor would be better to do simultaneously rather than separately. Thank goodness that refactoring in Rust is such a wonderful experience!

It depends on what you mean by "support graphs". If you mean support the abillity to build a GraphQL interface in front of it, yes that is already possible in a limited fashion, although there are no first-class relationship types yet.

I'm trying to release the first alpha of BonsaiDb. I'm wrapping up replacing OPAQUE with Argon2, in an effort to make upgrading less likely to cause issues in the future (given that OPAQUE is still a draft protocol). I still love OPAQUE and will bring it back in the future.

The repository seems abandoned; or maybe complete?

At work we use flume, which is another capable multi-producer, multi-consumer async-capable channel [1]. It's great for shuffling data between threads, as well as between async tasks, and between threads and async tasks. Basically any time you want to pieces of code to exchange data or signals without pesky shared state.

1: https://github.com/zesterer/flume

I also like flume, it has impressive performance (although not the best). More importantly, it's written only with safe rust. https://github.com/zesterer/flume

The most commonly suggested replacement for mspc is crossbeam-channel; flume is also relatively popular.

It's not officially deprecated, but the alternatives on crates.io are considered better. flume and crossbeam-channel feature less unsafe code and offer better performance. Benchmarks.

it's slow (checkout flume's benchmarks for example)

I would have done the same. I think, and I might be wrong, but the only other alternative, besides anything unsafe, would be to pass mutex back, but I am not sure this would be faster. Btw, I have not done testing, but you might want to look at Flume for your mpsc channels: https://github.com/zesterer/flume Flume, seems to be very fast mpsc implementation. I am planning to evaluate it for logging system.

For concurrency/parallelism, you launch at most 2 * CPU Cores, PIN them and use a fast broker to spread the task (like a ring buffer or an MPSC). But you keep linear scan, tight loops, SIMD friendly data, on each. You are not switching context that much, and instead, bet you will process the batch fast. (CPUs are fast today!)

https://crates.io/crates/flume and https://crates.io/crates/crossbeam-channel provide MPMC channels.