Taskflow VS emacs-async

For parallel programming nowadays, personally I reach for C++ Taskflow when I really care about performance, or a mix of core.async and running multiple load balanced instances when I’m doing more traditional web backend stuff in Clojure.

Also, you can take a look to good libraries. The problem is that very often libraries are heavily templated, so It could be hard. For example, I like the style of the Taskflow library, I think is very clear, is relatively small, while makes use of more advanced techniques: https://github.com/taskflow/taskflow

Cool. Thanks! How does it compare to taskflow?

I've not seen yet, but it's been a bit since I looked last, any evidence of being able to build a computation graph and "save" it to re-run on new inputs. Something like https://github.com/taskflow/taskflow

It seems you're describing something a generic parallel task framework. Check taskflow for a production ready example https://github.com/taskflow/taskflow/blob/master/

- https://github.com/taskflow/taskflow (I recommend to learn it first since its API and documentation are excellent)

If you want some sort of "job" system, where you submit items to a some sort of queue to be processed in parallel, try searching for a thread pool - there isn't one in the standard library, but there's about a million implementations online. There are more complicated versions of that idea, that describe computation as a directed acyclic graph, such as taskflow.

The tooling decisions affect engine design though. For example if you want to have visual representation of job graph as it happened in specific frame of interest you need to pass the information around about job relationships and output it to a tool of choice. For example see https://github.com/taskflow/taskflow

I am aware of DAGs of task based threading library like Taskflow and HPX however the benefit they have is not obvious to me, as the following sequential section depends on the parallel part being completed fully. If you want to suggest elaboration on the benefits of this approach would be welcome.

That is probably the simplest option possible; but if you need non-blocking evaluation, async package is definitely a better option.

You already can. Using https://github.com/jwiegley/emacs-async or https://github.com/chuntaro/emacs-promise.

IMO the best option currently is async by Wiegley. It will manage Emacs instances for you and do all the low-level synchronization and messaging for you, so you can work in higher level abstractions as if you are working with threads.

Have you checked the async package?

That is quite normal thing to do. Have you not seen Emacs Async? Take, a look, it is a useful thing. Or Emacs Request. Since Emacs does not have proper thread scheduler, that is the best next thing you can do.

There are ways to avoid this, have you tried e.g. https://github.com/jwiegley/emacs-async ?

As a note about the third video, and searching for backlinks; the volume, when you get there, might be a slow-down when you work with many small files, like searching for backlinks. Each note means a separate file access, search process, etc. It is much more efficient for computers to read one big file, then many small files, and then just use Emacs to search in that file. If you are a developer of Denote, you might wish to look at asynchronous processes or perhaps use Wigleys Async package to search for backlinks asynchronously.

Async process can do that. Have you checked async library by Wiegley? You can use another emacs process as a sort of clean interpreter thread similar to javascript workers.

That "99% of standard usage" is the kicker, isn't it? Those greybeards who always opposed multithreading since long ago tend to say that the remaining 1% of use cases is best done in an external process, ideally not even written in Emacs Lisp, so that the rest of the open source community can benefit, like the GNU Global you mention. I suppose if you still want that program to be written with Emacs Lisp, you could use async.el (https://github.com/jwiegley/emacs-async/) and there's finally an use-case for the threads: it'll be relatively safe to run those 16 threads only in the external Emacs-process.