ants VS conc

I remember doing something similar to OP recently. Goroutines also incur a bit of overhead (have to be GC'd and so on), and the same worker pool technique can be applied to them in much the same way, as seen in popular libraries like https://github.com/panjf2000/ants

The best example I can give you is https://github.com/nutsdb/nutsdb it’s great project that got me started, one thing one should know is Go is different “yep” so there’re some coding habits that may bite you in Go and the Go compiler won’t correct you, you wanna learn about optimizations, unsafe usage check out https://github.com/valyala/fasthttp (note this is deep the rabbit hole), wanna learn concurrency check out ants https://github.com/panjf2000/ants with a little aid from “Go by example” you’re good to go

conc seem similar to ant lib (https://github.com/panjf2000/ants). Is there any considerable difference between them?

But still, that looked hard to maintain, and I asked myself there has to be a better way to do this, just out of curiosity i googled and came across ants which seemed exactly right for this kind of functionality I wanted, so I converted the same function to use ants, and it became this:

You can try use this one https://github.com/panjf2000/ants We are using that as well for that purpose

Benchmarking was performed against existing golang threadpool implementations Ants and Gamma-Zero-Worker-Pool and unlimited goroutines

The readme of the project explains it very well: https://github.com/panjf2000/ants

Looks interesting, but what new concepts/features does it bring to the table compared to the already battletested ants library? Link: https://github.com/panjf2000/ants

It's a pity Go didn't have structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

There's a library for it: https://github.com/sourcegraph/conc

But this goes to one of the things I've been kind of banging on about languages, which is that if it's not in the language, or at least the standard library right at the beginning, sometimes it almost might as well not exist. Sometimes a new language can be valuable, even if it has no "new" language features, just to get a chance to reboot the standard library it has and push for patterns that older languages are theoretically capable of, but they just don't play well with any of the libraries in the language. Having it as a much-later 3rd party library just isn't good enough.

(In fact if I ever saw a new language start up and that was basically its pitch, I'd be very intrigued; it would show a lot of maturity in the language designer.)

What are possible use cases for this tool? Why would I want to find out the total runtime of all videos in a directory?

Also, you might wanna limit concurrency[0] instead of spawning many ffprobe instances at the ~same time.

[0]: https://github.com/sourcegraph/conc

In another note, ChatGPT suggests this shell command to do the same thing. It doesn't process files in parallel though.

    find . -name "*.mp4" -print0 | \

While virtual threads will be stable in Java 21, Structured Concurrency is still a preview feature. You probably won't see it in production anytime soon.

Preview features require a special flag when compiling and running them, and they won't run on newer versions of the JVM. I don't expect to see StructuredTaskScope in common production use before the next LTS version is out.

But it doesn't mean you cannot have structured concurrency before that. Even in language that mostly enforce Structured Concurrency like Kotlin, it's still a library feature. Even the original blog post which formulated this concept, described a library that implemented structured concurrency for Python[1]. You can pretty easily implement structured concurrency yourself by creating your own implementation of StructuredTaskScope, if you need it right now. You can even structured concurrency in C#[2] or Go[3].

[1] https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

[2] https://github.com/StephenCleary/StructuredConcurrency

[3] https://github.com/sourcegraph/conc

I'm also a fan of Conc for managing various different concurrency patterns -- don't create manual worker pools for locally distributed tasks if you can use Conc, etc.

How does this compare to conc?

I will probably be using more conc too now. Lots of great primitives for dealing with multiple functions returning the same types or errors etc.

You want a fast asynchronous development use a library I recommend https://github.com/sourcegraph/conc

A struct{} channel is the way standard way to implement this - yes. I don't think we're saying anything different? Here's a good implementation I used recently: https://github.com/sourcegraph/conc/blob/main/pool/pool.go