go-linq VS conc

There are also libraries like https://github.com/Jeffail/tunny or https://pkg.go.dev/go.uber.org/goleak or https://github.com/fatih/semgroup to help deal with concurrency limits and goroutine lifecycle management.

As the author of https://github.com/ahmetb/go-linq, it's hard to find adoption for libraries offering "syntactic sugar" in Go, as the language culture discourages those kind of abstractions and keeping the code straightforward.

This is a lot more possible now that Go has generics (as of 1.18).

I would probably never use these, as I find such libraries are a whole new domain-specific language to learn, and often don't make things much simpler anyway, but here are some libraries where people have done something like this:

* https://github.com/ahmetb/go-linq: modelled after LINQ, but created pre-generics so only recently added some generics features

So awhile back, there was a port of Linq https://github.com/ahmetb/go-linq that ended up giving up, since then it has been continued with https://github.com/szmcdull/glinq

I am aware of this that predates generics: https://github.com/ahmetb/go-linq/blob/master/groupby.go

Looks good, similar to https://github.com/sourcegraph/conc which we've been using for a while. Will give this a look.

JS concurrency is crap. It should be shot and buried in a lead coffin.

Debugging async code is pure hell. With Go, you have a normal debugger that can be used to step over the code. You can get normal stack traces for all threads if needed. There is a race detector that can catch most of unsynchronized object access.

With JS? You're on your fucking own. You can't find out the overall state of the system ("the list of all threads"), without getting deep into the guts of React or whatever framework you're using. Debugger is useless, as each `await` call drops you into the event loop. So pretty much every complicated non-trivial JS app ends up with _tons_ of race conditions, by depending on the order of async functions finishing.

Speaking of race conditions. Coming from classic multithreading and Go, I tried to use `Promise.race` to simulate the `select` statement. Turns out that in JS it is actually useless because it LEAKS MEMORY BY DESIGN: https://github.com/nodejs/node/issues/17469

Back to the article. It uses pre-generics Go. In more modern Go, you can use something like Conc to reduce the boilerplate: https://github.com/sourcegraph/conc?tab=readme-ov-file

Not speaking on whether the language should make this easier without an external library, but wouldn't https://github.com/sourcegraph/conc help in that scenario? It has context-aware and error-aware goroutine pools, seems like the exact fit for what you are trying to do. Although admittedly I dive too deep into your code.

It's a pity Go didn't have structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

There's a library for it: https://github.com/sourcegraph/conc

But this goes to one of the things I've been kind of banging on about languages, which is that if it's not in the language, or at least the standard library right at the beginning, sometimes it almost might as well not exist. Sometimes a new language can be valuable, even if it has no "new" language features, just to get a chance to reboot the standard library it has and push for patterns that older languages are theoretically capable of, but they just don't play well with any of the libraries in the language. Having it as a much-later 3rd party library just isn't good enough.

(In fact if I ever saw a new language start up and that was basically its pitch, I'd be very intrigued; it would show a lot of maturity in the language designer.)

What are possible use cases for this tool? Why would I want to find out the total runtime of all videos in a directory?

Also, you might wanna limit concurrency[0] instead of spawning many ffprobe instances at the ~same time.

[0]: https://github.com/sourcegraph/conc

In another note, ChatGPT suggests this shell command to do the same thing. It doesn't process files in parallel though.

    find . -name "*.mp4" -print0 | \

While virtual threads will be stable in Java 21, Structured Concurrency is still a preview feature. You probably won't see it in production anytime soon.

Preview features require a special flag when compiling and running them, and they won't run on newer versions of the JVM. I don't expect to see StructuredTaskScope in common production use before the next LTS version is out.

But it doesn't mean you cannot have structured concurrency before that. Even in language that mostly enforce Structured Concurrency like Kotlin, it's still a library feature. Even the original blog post which formulated this concept, described a library that implemented structured concurrency for Python[1]. You can pretty easily implement structured concurrency yourself by creating your own implementation of StructuredTaskScope, if you need it right now. You can even structured concurrency in C#[2] or Go[3].

[1] https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

[2] https://github.com/StephenCleary/StructuredConcurrency

[3] https://github.com/sourcegraph/conc

I'm also a fan of Conc for managing various different concurrency patterns -- don't create manual worker pools for locally distributed tasks if you can use Conc, etc.

How does this compare to conc?