oban VS transport-site

Oban, backed by PostgreSQL or SQLite, also provides a queue-based job processing system. Exq, on the other hand, is backed by Redis. It provides features similar to Flume, but without built-in rate limiting and batch processing capabilities.

In Elixir land Oban[0] uses Postgres as queue and seems to work quite well.

[0] - https://github.com/sorentwo/oban

I hear you on that, and can say that Postgres is incredibly capable at going beyond typical relational database workloads. One example are durable queues that are transactionally consistent with the rest of the database play a unique role in our architecture that would otherwise require more ceremony. More details here: https://getoban.pro

We are also working on shifting some workloads off of Postgres on to more appropriate systems as we scale, like logging. But we intentionally chose to minimize dependencies by pushing Postgres further to move faster, with migration plans ready as we continue to reach new levels of scale (e.g. using a dedicated log storage solution like elastic search or clickhouse).

You can actually have "background jobs" in very different ways in Elixir.

> I want background work to live on different compute capacity than http requests, both because they have very different resources usage

In Elixir, because of the way the BEAM works (the unit of parallelism is much cheaper and consume a low amount of memory), "incoming http requests" and related "workers" are not as expensive (a lot less actually) compared to other stacks (for instance Ruby and Python), where it is quite critical to release "http workers" and not hold the connection (which is what lead to the creation of background job tools like Resque, DelayedJob, Sidekiq, Celery...).

This means that you can actually hold incoming HTTP connections a lot longer without troubles.

A consequence of this is that implementing "reverse proxies", or anything calling third party servers _right in the middle_ of your own HTTP call, is usually perfectly acceptable (something I've done more than a couple of times, the latest one powering the reverse proxy behind https://transport.data.gouv.fr - code available at https://github.com/etalab/transport-site/tree/master/apps/un...).

As a consequence, what would be a bad pattern in Python or Ruby (holding the incoming HTTP connection) is not a problem with Elixir.

> because I want to have state or queues in front of background work so there's a well-defined process for retry, error handling, and back-pressure.

Unless you deal with immediate stuff like reverse proxying or cheap "one off async tasks" (like recording a metric), there also are solutions to have more "stateful" background works in Elixir, too.

A popular background job queue is https://github.com/sorentwo/oban (roughly similar to Sidekiq at al), which uses Postgres.

It handles retries, errors etc.

But it's not the only solution, as you have other tools dedicated to processing, such as Broadway (https://github.com/dashbitco/broadway), which handles back-pressure, fault-tolerance, batching etc natively.

You also have more simple options, such as flow (https://github.com/dashbitco/flow), gen_stage (https://github.com/elixir-lang/gen_stage), Task.async_stream (https://hexdocs.pm/elixir/1.12/Task.html#async_stream/5) etc.

It allows to use the "right tool for the job" quite easily.

It is also interesting to note there is no need to "go evented" if you need to fetch data from multiple HTTP servers: it can happen in the exact same process (even: in a background task attached to your HTTP server), as done here https://transport.data.gouv.fr/explore (if you zoom you will see vehicle moving in realtime, and ~80 data sources are being polled every 10 seconds & broadcasted to the visitors via pubsub & websockets).

Hi HN!

This is something I've been tinkering on for the past couple months. It's basically just an API/CLI for scheduling delayed or recurring jobs as HTTP requests.

I initially built it as a personal tool to save myself a bit of time on little side projects where I've needed scheduled/recurring alerts, but decided it could be a good opportunity to practice building out a nice landing page [0] and documentation [1]. And who knows, maybe someone else will find it useful ¯\_(ツ)_/¯

The tool relies heavily on Elixir's Oban [2] library for managing jobs, and Mintlify [3] for documentation. I also shamelessly stole most of the frontend design from Resend [4] because I'm a fan of the aesthetic and thought it would be good for my design chops to use their design as a guide. I also discovered Radix [5] UI while working on this, which ended up being immensely helpful for moving quickly on the frontend.

Anyways, I almost certainly spent a bit too much time on small UX details that are most likely utterly inconsequential, but it was a fun exercise in polish :)

All feedback is welcome!

[0] https://www.booper.dev/

[1] https://docs.booper.dev/

[2] https://github.com/sorentwo/oban

[3] https://mintlify.com/

[4] https://resend.com/

[5] https://www.radix-ui.com/

Idk about pgagent but any table is a resilient queue with the multiple locks available in pg along with some SELECT pg_advisory_lock or SELECT FOR UPDATE queries, and/or LISTEN/NOTIFY.

Several bg job libs are built around native locking functionality

> Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of schema.rb, and LISTEN/NOTIFY to reduce queuing latency.

https://github.com/bensheldon/good_job

> |> lock("FOR UPDATE SKIP LOCKED")

https://github.com/sorentwo/oban/blob/8acfe4dcfb3e55bbf233aa...

> Bad code in a specific part of the codebase bringing down the whole app, as in our November incident.

This is a non-issue if you're using a Elixir/Erlang monolith given its fault tolerant nature.

The noisy neighbour issue (resource hogging) is still something you need to manage though. If you use something like Oban[1] (for background job queues and cron jobs), you can set both local and global limits. Local being the current node, and global the cluster.

Operating in a shared cluster (vs split workload deployments) give you the benefit of being much more efficient with your hardware. I've heard many stories of massive infra savings due to moving to an Elixir/Erlang system.

1. https://github.com/sorentwo/oban

You can actually have "background jobs" in very different ways in Elixir.

> I want background work to live on different compute capacity than http requests, both because they have very different resources usage

In Elixir, because of the way the BEAM works (the unit of parallelism is much cheaper and consume a low amount of memory), "incoming http requests" and related "workers" are not as expensive (a lot less actually) compared to other stacks (for instance Ruby and Python), where it is quite critical to release "http workers" and not hold the connection (which is what lead to the creation of background job tools like Resque, DelayedJob, Sidekiq, Celery...).

This means that you can actually hold incoming HTTP connections a lot longer without troubles.

A consequence of this is that implementing "reverse proxies", or anything calling third party servers _right in the middle_ of your own HTTP call, is usually perfectly acceptable (something I've done more than a couple of times, the latest one powering the reverse proxy behind https://transport.data.gouv.fr - code available at https://github.com/etalab/transport-site/tree/master/apps/un...).

As a consequence, what would be a bad pattern in Python or Ruby (holding the incoming HTTP connection) is not a problem with Elixir.

> because I want to have state or queues in front of background work so there's a well-defined process for retry, error handling, and back-pressure.

Unless you deal with immediate stuff like reverse proxying or cheap "one off async tasks" (like recording a metric), there also are solutions to have more "stateful" background works in Elixir, too.

A popular background job queue is https://github.com/sorentwo/oban (roughly similar to Sidekiq at al), which uses Postgres.

It handles retries, errors etc.

But it's not the only solution, as you have other tools dedicated to processing, such as Broadway (https://github.com/dashbitco/broadway), which handles back-pressure, fault-tolerance, batching etc natively.

You also have more simple options, such as flow (https://github.com/dashbitco/flow), gen_stage (https://github.com/elixir-lang/gen_stage), Task.async_stream (https://hexdocs.pm/elixir/1.12/Task.html#async_stream/5) etc.

It allows to use the "right tool for the job" quite easily.

It is also interesting to note there is no need to "go evented" if you need to fetch data from multiple HTTP servers: it can happen in the exact same process (even: in a background task attached to your HTTP server), as done here https://transport.data.gouv.fr/explore (if you zoom you will see vehicle moving in realtime, and ~80 data sources are being polled every 10 seconds & broadcasted to the visitors via pubsub & websockets).

The Erlang "telemetry" module can even be used "applicatively" with good uses.

An example of that is a "caching reverse proxy" in use at https://transport.data.gouv.fr/, which helps protect data producers servers from excessive load from data consumers (which we manage for them).

"External" requests come to us (via proxy.transport.data.gouv.fr) and if the response is not in cache (for a configured TTL), we forward the request as an "internal" request to the target (3rd party) server.

The proxy uses "telemetry" to generate events for these internal/external queries, without knowing how they will be consumed (https://github.com/etalab/transport-site/blob/master/apps/un...).

The main app receives those events and does what it wants, in this case upserting metrics in the database (https://github.com/etalab/transport-site/blob/master/apps/tr...).

As a result, it's easy to use LiveView to create a real-time dashboard to show these metrics (https://jumpshare.com/s/rdGtTa47CzlcgOz4jsOI - low traffic at time of posting) or generate stats for the data producers which are behind the proxy.

It's all quite natural to do because it's really baked into the runtime!

We're using on the French national access point to transportation data (http://transport.data.gouv.fr/?locale=en, source code at https://github.com/etalab/transport-site) and we're not going to switch to another stack ^_^.

Plenty of use-cases are made easy by the stack, from regular web work, API (https://transport.data.gouv.fr/swaggerui) to proxying (https://github.com/etalab/transport-site/tree/master/apps/un...) to real-time maps (https://transport.data.gouv.fr/explore), clustered maps (https://transport.data.gouv.fr/explore/gtfs-stops), XML queries building (https://transport.data.gouv.fr/tools/siri-querier?endpoint_u...)...

The maintenance story is very good too.

With ML being added (Nx/Axon etc), and mobile apps being in the works (LiveViewNative), it has become my everyday language & stack.

> Yes, this is as bad as it looks: “success” isn’t even part of the schema. It’s in the examples, but not the actual schema definition.

Having gone through a pretty heavy overhaul of an "OpenAPI" (full code in Elixir at https://github.com/etalab/transport-site/pull/3351), I stumbled on that exact type of problem!

At the scale of nginx, having automatic verification that the examples (and the output of the API in general) match the specification would be great.

At our scale, here is what really helped me go through the rework (and ensure we do not regress too easily):

- setting additionalProperties to "false" to detect key field "rot"

- using "required: [x,y,z]" on everything, and by default specify "all the property keys", with an opt-out (so that each time a developer adds a field later, it is considered mandatory, unless otherwise specified)

- use tooling during the tests: "assert_schema" (with OpenAPISpex) to ensure our API endpoints responses pass the spec (additionalProperties: false helps ensure we get an exception in case of key field rot, again!)

- even more useful: crawl our production most important endpoints and tweak the spec until everything is green (an example of useful use of Task.async_stream in Elixir, by the way) https://github.com/etalab/transport-site/pull/3351/files#dif...

It can be super frustrating for users to live with the uncertainty of the response of an API for sure, and I was happy to discover the Elixir tooling (OpenAPISpex in particular) worked so nicely once I understood what I had to do.

Very convenient to parallelise HTTP queries (and without the need to go “evented”).

One recent example where I assert that API responses match our OpenAPI specifications here, for the curious:

https://github.com/etalab/transport-site/pull/3351/files#dif...

I've been a big fan of "Mix.install" since it was released!

I believe this made me use Elixir instead of Ruby (my natural scripting language) more and more.

I use it on a regular basis for my work on https://transport.data.gouv.fr/ (which is Elixir-based and open-source), as one can see at https://github.com/etalab/transport-site/tree/master/scripts.

What I like the most is that it helps me start ideas, experiments & data analysis with their own set of dependencies (without much care of "will this impact the main application?"), store those experiments in the same repo at the main application, and maybe later promote some of those experiments to the main application source code.

Concrete examples include:

I'm one of the 3 maintainers of the French open-data transportation access point (at https://transport.data.gouv.fr/?locale=en), which runs on top of Elixir & Phoenix.

I'm not a big fan of surveys and I'm not sure I've responded to this one actually, but: my current feeling around Phoenix is that it will be my go-to framework for anything web related for the years to come, but also Elixir will be, for larger topics (ML, embedded, data-viz, scripting, etc).

There are a number of important points (to my taste) that comes out of my use of Elixir & Phoenix :

- the maintenance story has been quite good (maintenance work is one of my main line of work since year 2k, I have used many stacks, including Java, .Net, RubyOnRails etc). Upgrades and refactoring & moving code around has been quite easy overall, once you get the stuff right (using Mox etc).

- Phoenix, Ecto & Elixir are a good "generalist web framework" for a lot of use

- Compared to some stacks (non-evented Ruby & Python, for instance), having a part of your app work as a proxy for domain-specific uses, is not a problem, and does not even require you to split that part from the main app (example at https://github.com/etalab/transport-site/tree/master/apps/un..., where we proxy/cache GTFS-RT protobuf & SIRI XML feeds behind rules, adding rate-limiting & hiding target server authentication schemes)

- Similarly, anything real-time is at least initially trivial. For instance, creating this real-time view of moving vehicles (https://transport.data.gouv.fr/explore) with 70+ GTFS-RT feeds has been quite easy (PR at https://github.com/etalab/transport-site/pull/2224). The server side is like "EventMachine done right", and the live updates are broadcasted to each client via web socket. No extra server (e.g. AnyCable) or anything is needed.

From a perf/ops POV, the amount of topics you do not even have to think about is quite important, which is very nice.

Overall I can see myself sticking to this stack for the next 15 years or so, without much worries (I've been coding since 1984 as a kid, so I have a good idea to know when to keep a framework around or not :P)

You can check this gov website : https://transport.data.gouv.fr/

transport.data.gouv.fr, the transport part of the french open data portal. Source code here: https://github.com/etalab/transport-site/.

Stripe, Shopify and GitHub are still largely relying on Ruby as far as I know (and while there are reimplementations of some services in some langages, I don't think C++ or Java have been publicly mentioned often).

That said, I think one important asset of Elixir (in my view) is not the raw performance, but the overall "total cost of ownership" (including maintenance work, and software is a lot of maintenance).

Being able to simplify architectures is something that is nicely done with Elixir, actually.

If a part of an app needs to play a "proxy" role (like here https://github.com/etalab/transport-site/tree/master/apps/un...), then I just add a component to the app, and I can keep the incoming connections under hand, all while issuing external queries etc.

If there is a need to do some rich interactive dashboards, I can use LiveView.

If I need to demonstrate some simple stuff, I can use "Mix.install" to create one-off scripts.

If I need to do more data-science, I can tap into LiveBook, VegaLite etc.

All this with a small team or as a solo developer, is quite great, much more than raw performance, in my eyes.