heka VS conc

The channel-focused approach to stream processing reminds me of Heka [0]. It was a contemporary of Samza and Heron, and it was fairly prominent in the early Go ecosystem (maybe 10 years ago). As I recall it, quite foggily and quite a long while later, one of the final nails in Heka's coffin was that channel throughput didn't scale well. Do you have benchmarks for Rill, or is it not intended for high-throughput use cases?

[0]: https://github.com/mozilla-services/heka

> Actually much closer than anything I saw in other mainstream languages eg Java or Go.

https://github.com/sourcegraph/conc

Looks good, similar to https://github.com/sourcegraph/conc which we've been using for a while. Will give this a look.

JS concurrency is crap. It should be shot and buried in a lead coffin.

Debugging async code is pure hell. With Go, you have a normal debugger that can be used to step over the code. You can get normal stack traces for all threads if needed. There is a race detector that can catch most of unsynchronized object access.

With JS? You're on your fucking own. You can't find out the overall state of the system ("the list of all threads"), without getting deep into the guts of React or whatever framework you're using. Debugger is useless, as each `await` call drops you into the event loop. So pretty much every complicated non-trivial JS app ends up with _tons_ of race conditions, by depending on the order of async functions finishing.

Speaking of race conditions. Coming from classic multithreading and Go, I tried to use `Promise.race` to simulate the `select` statement. Turns out that in JS it is actually useless because it LEAKS MEMORY BY DESIGN: https://github.com/nodejs/node/issues/17469

Back to the article. It uses pre-generics Go. In more modern Go, you can use something like Conc to reduce the boilerplate: https://github.com/sourcegraph/conc?tab=readme-ov-file

Not speaking on whether the language should make this easier without an external library, but wouldn't https://github.com/sourcegraph/conc help in that scenario? It has context-aware and error-aware goroutine pools, seems like the exact fit for what you are trying to do. Although admittedly I dive too deep into your code.

It's a pity Go didn't have structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

There's a library for it: https://github.com/sourcegraph/conc

But this goes to one of the things I've been kind of banging on about languages, which is that if it's not in the language, or at least the standard library right at the beginning, sometimes it almost might as well not exist. Sometimes a new language can be valuable, even if it has no "new" language features, just to get a chance to reboot the standard library it has and push for patterns that older languages are theoretically capable of, but they just don't play well with any of the libraries in the language. Having it as a much-later 3rd party library just isn't good enough.

(In fact if I ever saw a new language start up and that was basically its pitch, I'd be very intrigued; it would show a lot of maturity in the language designer.)

What are possible use cases for this tool? Why would I want to find out the total runtime of all videos in a directory?

Also, you might wanna limit concurrency[0] instead of spawning many ffprobe instances at the ~same time.

[0]: https://github.com/sourcegraph/conc

In another note, ChatGPT suggests this shell command to do the same thing. It doesn't process files in parallel though.

    find . -name "*.mp4" -print0 | \

While virtual threads will be stable in Java 21, Structured Concurrency is still a preview feature. You probably won't see it in production anytime soon.

Preview features require a special flag when compiling and running them, and they won't run on newer versions of the JVM. I don't expect to see StructuredTaskScope in common production use before the next LTS version is out.

But it doesn't mean you cannot have structured concurrency before that. Even in language that mostly enforce Structured Concurrency like Kotlin, it's still a library feature. Even the original blog post which formulated this concept, described a library that implemented structured concurrency for Python[1]. You can pretty easily implement structured concurrency yourself by creating your own implementation of StructuredTaskScope, if you need it right now. You can even structured concurrency in C#[2] or Go[3].

[1] https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

[2] https://github.com/StephenCleary/StructuredConcurrency

[3] https://github.com/sourcegraph/conc

I'm also a fan of Conc for managing various different concurrency patterns -- don't create manual worker pools for locally distributed tasks if you can use Conc, etc.