crossbeam VS rayon

Crossbeam isn't async[0]. It can multiplex with itself (via the `select!` macro), but not with anything else.

[0]: https://github.com/crossbeam-rs/crossbeam/issues/896

Why use this or Flume over the crossbeam multi-producer multi-consumer channel [1]? I thought crossbeam was well-regarded and pretty much the unofficial standard library for sync code.

https://github.com/crossbeam-rs/crossbeam/tree/master/crossb...

Shooting from the hip, crossbeam might be a good candidate for understanding the thread safety aspects of Rust. I kind of feel like this is probably "too big" of a project if you're just learning, but I can't think of something smaller off the top of my head that would be suitable.

The crossbeam crate offers a powerful alternative to standard channels with support for the Select operation, timeouts, and more.

Deadlocks and leaks are easy as other languages.

Ouch, didn’t know about https://github.com/crossbeam-rs/crossbeam/issues/821, thanks for pointing that out, that’s a big update for me!

The last option I can think of is using two threads (like above) and epoch GC instead of a lock (i.e. using crossbeam-epoch). But I don't have enough experience with this to say anything about it.

rayon: Async CPU runtime for parallelism.

In the quest for more efficient solutions, the ONNX runtime emerged as a beacon of performance. The decision to transition from Typescript to Rust was an unconventional yet pivotal one. Driven by Rust's robust parallel processing capabilities using Rayon and seamless integration with ONNX through the ort crate, Repo-Query unlocked a realm of unparalleled efficiency. The result? A transformation from sluggish processing to, I have to say it, blazing-fast performance.

(see https://github.com/rayon-rs/rayon/tree/master/src/iter/plumbing)

Sequentially, generating tree-sitter AST for each file and querying for the links of each file takes around 2.3 seconds. However, I randomly remembered this crate rayon, and I decided to test it. It ended up improving the performance (just by changing 2 lines of code) to 200-300ms by parallelizing the iterators and tree-sitter queries. MAJOR.

Now if you really want to use Rust, you can rewrite only the part that are slowing down your consumer. It's easy by using Py03 and maturin. Maybe also rayon to parallelize.

Maybe rayon could make some loops there and there faster if needed.

One of the others understated pros of rewriting some parts in Rust, it's that you can parallelize easily with Rayon[0]

[0] -- https://github.com/rayon-rs/rayon

On that topic, have you heard about our lord and savior rayon? 😊

> He spoke of the potential for functional languages to provide a significant, intrinsic advantage when it comes to parallel computing.

> (...)

> If that were true, you'd expect that the many existing functional programming languages would have already satisfied this need. But in my opinion, they haven't

Well there is https://futhark-lang.org/ - it runs on the GPU, and is awesome.

On the CPU side, I think that Rust plus https://github.com/rayon-rs/rayon was a huge breakthrough on writing parallel programs using both functional and imperative programming, and future languages should learn from its successes. The ownership system & the borrow checker, plus other type-level features like the Send and Sync traits, were essential to enable sharing read-only data between threads without synchronization, or sharing read-write data with synchronization, all checked at compile time for data races (which is a huge problem to solve, and is something that neither Go nor Java protects against at compile time)

Indeed Futhark shares a key feature with Rust: it uses uniqueness types to enable in-place updates, which is kind like a limited form of Rust ownership: if you are the sole user of some memory, you can update it and other code will be none the wiser. This kind of thing is very important to build functional programs that are performant in practice.