cff VS rill

Sourcegraph Conc is broadly similar in providing pool helpers, but doesn't provide the same fine grained batching options: https://github.com/sourcegraph/conc

Uber CFF does code generation, and has more of a focus on readability and complex dependency chains: https://github.com/uber-go/cff

"The amount of APIs in the world that require concurrency within a single request scope to meet latency needs approaches zero."

I've never worked on any non-Rails API where this was true. The Ruby community keeps telling me this, but in languages where concurrency is well supported, it gets used everywhere to a great extent. I obviously don't have hard data to support this, but your claim seems pretty far fetched to be honest.

"In practice, you don’t make that db call until the auth request is done and the user is verified"

Of course you optimistically do any idempotent DB operations while waiting for auth to succeed if you care about latency.

"you don’t make the outgoing api call until you already have the results of the db call, because you need your data to form the outgoing request"

These dependencies of course exist, but so do parts of the graph where they do not. You might want to make 2 or 5 outgoing calls based on your DB query and have to wait for 2 out of these 5 to make another DB query. This is so common that there are libraries like https://github.com/uber-go/cff to explicitly model those dependency graphs, visualize them and resolve them at runtime.

Wow, that’s some seriously sophisticated stuff - it’s not that often you see a heap used in typical production code (outside of libraries)!

Your first example definitely gives me merge-sort vibes - a really clean way to keep things ordered across multiple sources. The second and third scenarios are a bit beyond what I’ve tackled so far, but super interesting to read about.

This also reminded me of a WIP PR I drafted for rill (probably too niche, so I’m not sure I’ll ever merge it). It implements a channel buffer that behaves like a heap - basically a fixed-size priority queue where re-prioritization only happens for items that pile up due to backpressure. Maybe some of that code could be useful for your future use cases: https://github.com/destel/rill/pull/50

To simplify the worker implementation, we'll use rill, a concurrency toolkit I’ve built. It provides utilities like rill.Batch, which groups incoming items into batches with a timeout (exactly as described earlier), and rill.ForEach, which helps control the level of concurrency when processing those batches. Thanks to these tools, the worker implementation becomes remarkably compact.