readerwriterqueue VS HPX

I have looked at the commonly recommended - https://github.com/cameron314/readerwriterqueue - https://github.com/rigtorp/MPMCQueue

The PREEMPT error looks like its a C module but I cannot tell from he little snippet of the log - you may do better to move the *arr and Synchthing into Docker containers to keep the OS cleaner.

For audio DSP in a Qt app I have good success with using a lock-free queue (e.g. https://github.com/cameron314/readerwriterqueue) to communicate from / to the UI threads. Code more-or-less looks like

    connect(model, &ModelObject::somethingChanged, [=] {

We recently removed readerwriterqueue from our project due to issues and replaced with simple locking queue (and got better performance in the result). Your code looks interesting, but we can't use it due to C++17 usage.

https://github.com/STEllAR-GROUP/hpx Modern C++ concepts incorporated in a threading library. Lots of useful techniques used in there and we are trying to keep our code base very tidy. Feel free to chime in our libera channel #ste||ar if you have any questions.

https://github.com/STEllAR-GROUP/hpx Welcoming community + we have been part of GSoC for 4-5 years now so feel free to apply there when it opens ;)

You definitely do not want to learn Boost, trust me. The cudatoolkit is fine, HPX is great, so are Dask, and Ray. I do not recommend MPI unless those computers you have use InfiniBand.

I found this interesting library that might fit your goal.

I personally like the ideas in Parallelism v2 TS, which is available in for libstdc++ 11 onwards. The reference implementation is a library named Vc (afaik Vc is the most popular SIMD library for C++), and this has also been implemented in recent versions of HPX.

I am aware of DAGs of task based threading library like Taskflow and HPX however the benefit they have is not obvious to me, as the following sequential section depends on the parallel part being completed fully. If you want to suggest elaboration on the benefits of this approach would be welcome.

Github: https://github.com/STEllAR-GROUP/hpx

We're working on something similar, but based on sender/receiver in HPX (a lightweight threading runtime) and DLA-Future (distributed linear algebra currently based on (HPX) futures; based on sender/receiver in the future). With senders-as-awaitables this would also get you coroutine support for asynchronous MPI calls for free. We don't have that yet, but it's planned. In the meantime libunifex should be able to fill in the gaps.