changelog.com VS oban

Another https://dagger.io fan here. Have been using it since late 2021 to continuously deploy a Phoenix app to Fly.io: https://github.com/thechangelog/changelog.com/pull/395. Every commit goes into production.

This is what the GHA workflow currently looks like: https://github.com/thechangelog/changelog.com/blob/c7b8a57b2...

FWIW, you can see how everything fits together in this architecture diagram: https://github.com/thechangelog/changelog.com/blob/master/IN...

I really like the work that you're doing Thomas, this is the right approach. FWIW, https://fly.io/blog/carving-the-scheduler-out-of-our-orchest... is one of my favourite posts on your blog.

For everyone else reading this, we have been running https://changelog.com on Fly.io since April 2022. This is what our architecture currently looks like: https://github.com/thechangelog/changelog.com/blob/master/IN...

After 15 months & more than 100 million requests served by our Phoenix + PostgreSQL app running on Fly.io, I would be hard pressed to find a reason to complain.

Changelog.com

I see this criticism a lot but I don't think it has anything to do with macros specifically and more so to do with lack of familiarity with Elixir. I've felt the same way about Django being magic because I had trouble following the class hierarchy. It makes a lot more sense now because I'm more familiar with Python and Django. But even today I'll be looking deeper at something and ask WTF it's doing. In that respect, Elixir codebases are easier to me. The module depth seems "shallower" and I don't have to disambiguate between what behavior is caused by class inheritance or an imported function.

When I first tried to use Elixir several years ago Ecto.Schema [0] seemed complex and magical, but then I came to realize it's just converting module attributes to runtime code. There is not really that much complex macro logic going on.

>This also leads to cryptic errors where you get an error in non existant lines of code.

When was the last time you used Elixir? This isn't a problem I can recall having in the last 4 years or so of using Elixir.

>following the control flow in Phoenix is like a maze because of all the macro substitutions.

Can you clarify what you mean by this? A specific case as to where this happened for you would help. Phoenix's use of macros is actually pretty light [1] except for some very low level stuff. You can even see how frequently a developer will use macros in Phoenix by searching `__using__` in the codebase [2]. It's not used as much as people think. The majority is for views and controllers and only to provide a very thin layer of support on top of your regular use of code. As an example, the "macro magic" in Phoenix.Controller is just handling some basics for giving a layout and view to Plug and handling fallback actions for exceptions. You could do the plug calls manually and I think it would be safe to not use any macros in your controller code.

Another familiarity issue with the language (and any language really) is understanding what is meaningful in a stack trace and what isn't. And the likely cause of the error in the first place. Is it syntax? Is it mistyping a variable? Is a function just used improperly? (wtf is init_p_do_apply and why does it show up in every stacktrace?) You're juggling all these different issues - learning a new paradigm, a new syntax, not knowing how to extend things. It's obviously going to be a little overwhelming and, if not strictly required, we might just pick a different language that we're more familiar with.

[0]: https://github.com/elixir-ecto/ecto/blob/master/lib/ecto/sch...

[1]: https://github.com/thechangelog/changelog.com/blob/master/li... (not mine, I just go here to show the most frequent use of macros in Phoenix)

[2]: https://github.com/phoenixframework/phoenix/search?q=__using...

changelog.com used to be WordPress, then became a Phoenix app because it needed features that were hacky to implement & then manage in WP. It's more of a podcasting platform these days rather than a CMS.

The code in this repo tells the truth about what it is, and even shows how it works: https://github.com/thechangelog/changelog.com

Oban, backed by PostgreSQL or SQLite, also provides a queue-based job processing system. Exq, on the other hand, is backed by Redis. It provides features similar to Flume, but without built-in rate limiting and batch processing capabilities.

In Elixir land Oban[0] uses Postgres as queue and seems to work quite well.

[0] - https://github.com/sorentwo/oban

I hear you on that, and can say that Postgres is incredibly capable at going beyond typical relational database workloads. One example are durable queues that are transactionally consistent with the rest of the database play a unique role in our architecture that would otherwise require more ceremony. More details here: https://getoban.pro

We are also working on shifting some workloads off of Postgres on to more appropriate systems as we scale, like logging. But we intentionally chose to minimize dependencies by pushing Postgres further to move faster, with migration plans ready as we continue to reach new levels of scale (e.g. using a dedicated log storage solution like elastic search or clickhouse).

You can actually have "background jobs" in very different ways in Elixir.

> I want background work to live on different compute capacity than http requests, both because they have very different resources usage

In Elixir, because of the way the BEAM works (the unit of parallelism is much cheaper and consume a low amount of memory), "incoming http requests" and related "workers" are not as expensive (a lot less actually) compared to other stacks (for instance Ruby and Python), where it is quite critical to release "http workers" and not hold the connection (which is what lead to the creation of background job tools like Resque, DelayedJob, Sidekiq, Celery...).

This means that you can actually hold incoming HTTP connections a lot longer without troubles.

A consequence of this is that implementing "reverse proxies", or anything calling third party servers _right in the middle_ of your own HTTP call, is usually perfectly acceptable (something I've done more than a couple of times, the latest one powering the reverse proxy behind https://transport.data.gouv.fr - code available at https://github.com/etalab/transport-site/tree/master/apps/un...).

As a consequence, what would be a bad pattern in Python or Ruby (holding the incoming HTTP connection) is not a problem with Elixir.

> because I want to have state or queues in front of background work so there's a well-defined process for retry, error handling, and back-pressure.

Unless you deal with immediate stuff like reverse proxying or cheap "one off async tasks" (like recording a metric), there also are solutions to have more "stateful" background works in Elixir, too.

A popular background job queue is https://github.com/sorentwo/oban (roughly similar to Sidekiq at al), which uses Postgres.

It handles retries, errors etc.

But it's not the only solution, as you have other tools dedicated to processing, such as Broadway (https://github.com/dashbitco/broadway), which handles back-pressure, fault-tolerance, batching etc natively.

You also have more simple options, such as flow (https://github.com/dashbitco/flow), gen_stage (https://github.com/elixir-lang/gen_stage), Task.async_stream (https://hexdocs.pm/elixir/1.12/Task.html#async_stream/5) etc.

It allows to use the "right tool for the job" quite easily.

It is also interesting to note there is no need to "go evented" if you need to fetch data from multiple HTTP servers: it can happen in the exact same process (even: in a background task attached to your HTTP server), as done here https://transport.data.gouv.fr/explore (if you zoom you will see vehicle moving in realtime, and ~80 data sources are being polled every 10 seconds & broadcasted to the visitors via pubsub & websockets).

Hi HN!

This is something I've been tinkering on for the past couple months. It's basically just an API/CLI for scheduling delayed or recurring jobs as HTTP requests.

I initially built it as a personal tool to save myself a bit of time on little side projects where I've needed scheduled/recurring alerts, but decided it could be a good opportunity to practice building out a nice landing page [0] and documentation [1]. And who knows, maybe someone else will find it useful ¯\_(ツ)_/¯

The tool relies heavily on Elixir's Oban [2] library for managing jobs, and Mintlify [3] for documentation. I also shamelessly stole most of the frontend design from Resend [4] because I'm a fan of the aesthetic and thought it would be good for my design chops to use their design as a guide. I also discovered Radix [5] UI while working on this, which ended up being immensely helpful for moving quickly on the frontend.

Anyways, I almost certainly spent a bit too much time on small UX details that are most likely utterly inconsequential, but it was a fun exercise in polish :)

All feedback is welcome!

[0] https://www.booper.dev/

[1] https://docs.booper.dev/

[2] https://github.com/sorentwo/oban

[3] https://mintlify.com/

[4] https://resend.com/

[5] https://www.radix-ui.com/

Idk about pgagent but any table is a resilient queue with the multiple locks available in pg along with some SELECT pg_advisory_lock or SELECT FOR UPDATE queries, and/or LISTEN/NOTIFY.

Several bg job libs are built around native locking functionality

> Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of schema.rb, and LISTEN/NOTIFY to reduce queuing latency.

https://github.com/bensheldon/good_job

> |> lock("FOR UPDATE SKIP LOCKED")

https://github.com/sorentwo/oban/blob/8acfe4dcfb3e55bbf233aa...

> Bad code in a specific part of the codebase bringing down the whole app, as in our November incident.

This is a non-issue if you're using a Elixir/Erlang monolith given its fault tolerant nature.

The noisy neighbour issue (resource hogging) is still something you need to manage though. If you use something like Oban[1] (for background job queues and cron jobs), you can set both local and global limits. Local being the current node, and global the cluster.

Operating in a shared cluster (vs split workload deployments) give you the benefit of being much more efficient with your hardware. I've heard many stories of massive infra savings due to moving to an Elixir/Erlang system.

1. https://github.com/sorentwo/oban