gen_stage VS rustic_result

You can actually have "background jobs" in very different ways in Elixir.

> I want background work to live on different compute capacity than http requests, both because they have very different resources usage

In Elixir, because of the way the BEAM works (the unit of parallelism is much cheaper and consume a low amount of memory), "incoming http requests" and related "workers" are not as expensive (a lot less actually) compared to other stacks (for instance Ruby and Python), where it is quite critical to release "http workers" and not hold the connection (which is what lead to the creation of background job tools like Resque, DelayedJob, Sidekiq, Celery...).

This means that you can actually hold incoming HTTP connections a lot longer without troubles.

A consequence of this is that implementing "reverse proxies", or anything calling third party servers _right in the middle_ of your own HTTP call, is usually perfectly acceptable (something I've done more than a couple of times, the latest one powering the reverse proxy behind https://transport.data.gouv.fr - code available at https://github.com/etalab/transport-site/tree/master/apps/un...).

As a consequence, what would be a bad pattern in Python or Ruby (holding the incoming HTTP connection) is not a problem with Elixir.

> because I want to have state or queues in front of background work so there's a well-defined process for retry, error handling, and back-pressure.

Unless you deal with immediate stuff like reverse proxying or cheap "one off async tasks" (like recording a metric), there also are solutions to have more "stateful" background works in Elixir, too.

A popular background job queue is https://github.com/sorentwo/oban (roughly similar to Sidekiq at al), which uses Postgres.

It handles retries, errors etc.

But it's not the only solution, as you have other tools dedicated to processing, such as Broadway (https://github.com/dashbitco/broadway), which handles back-pressure, fault-tolerance, batching etc natively.

You also have more simple options, such as flow (https://github.com/dashbitco/flow), gen_stage (https://github.com/elixir-lang/gen_stage), Task.async_stream (https://hexdocs.pm/elixir/1.12/Task.html#async_stream/5) etc.

It allows to use the "right tool for the job" quite easily.

It is also interesting to note there is no need to "go evented" if you need to fetch data from multiple HTTP servers: it can happen in the exact same process (even: in a background task attached to your HTTP server), as done here https://transport.data.gouv.fr/explore (if you zoom you will see vehicle moving in realtime, and ~80 data sources are being polled every 10 seconds & broadcasted to the visitors via pubsub & websockets).

I think if the bot gets a lot of users or streams being tracked you might run into issues with how sync_streams works, you might want to look at something like GenStage or maybe Oban to send messages in batches. In a similar note you'd want to do the same thing for cases where you are doing Repo.all.

Self promotion moment:

If you enjoy the Result/Either type and API in Rust, I made this project just for this: https://github.com/linkdd/rustic_result

I also made https://github.com/linkdd/rustic_maybe/tree/main for an Option/Maybe type.

NB: Those are not types, but I'm waiting for set theoretic types to update those libs :)

It's missing the Erlang/Elixir pattern of returning a tuple `{:ok, T}` or `{:error, E}`, where we can then use pattern matching, or `with` expressions, etc...

To be fair, it is very similar to a `Result` type, which is why I made this library a while ago: https://github.com/linkdd/rustic_result

I intend to add a pipeline operator similar to Elixir, to be used with an std::result module which will provide functions to ease writing such complex code. I may take inspiration on an Elixir library I wrote a while ago: https://github.com/linkdd/rustic_result

A while ago, I made this library https://github.com/linkdd/rustic_result

I also made a library for working with `{:ok, value}` and `{:error, reason}` in Elixir: https://github.com/linkdd/rustic_result

Thanks to the pipeline operator and pattern matching, it makes pretty easy to read pipelines. It does not completely replace the with statement (that was not the point) but it simplified a lot of code.