Que VS oban

Compare Que vs oban and see what are their differences.

Que

A Ruby job queue that uses PostgreSQL's advisory locks for speed and reliability. (by que-rb)

oban

πŸ’Ž Robust job processing in Elixir, backed by modern PostgreSQL and SQLite3 (by sorentwo)
InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
Que oban
10 27
2,287 3,056
0.2% -
5.6 9.3
6 days ago 4 days ago
Ruby Elixir
MIT License Apache License 2.0
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.

Que

Posts with mentions or reviews of Que. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-09-24.
  • Choose Postgres Queue Technology
    17 projects | news.ycombinator.com | 24 Sep 2023
    > Can you define "low throughput"?

    <1000 messages per minute

    Not saying SKIP LOCKED can't work with that many. But you'll probably want to do something better.

    FWIW, Que uses advisory locks [1]

    [1] https://github.com/que-rb/que

  • Introducing tobox: a transactional outbox framework
    2 projects | /r/ruby | 29 Apr 2023
    Probably worth mentioning that aside from delayed_job there are at least two more modern alternatives backed by the DB: Que and good_job.
  • Sidekiq jobs in ActiveRecord transactions
    2 projects | /r/rails | 27 Apr 2023
    Good article. Sidekiq is a good, well respected too. However if you are starting out I would recommend not using it, and instead choosing a DB based queue system. We have great success with que, but there are others like good_job.
  • SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
    16 projects | news.ycombinator.com | 11 Apr 2023
    (not sure why this comment was dead, I vouched for it)

    There are a lot of ways to implement a queue in an RDBMS and a lot of those ways are naive to locking behavior. That said, with PostgreSQL specifically, there are some techniques that result in an efficient queue without locking problems. The article doesn't really talk about their implementation so we can't know what they did, but one open source example is Que[1]. Que uses a combination of advisory locking rather than row-level locks and notification channels to great effect, as you can read in the README.

    [1]: https://github.com/que-rb/que

  • Delayed Job vs. Sidekiq: Which Is Better?
    13 projects | news.ycombinator.com | 16 Feb 2022
    https://github.com/que-rb/que

    This one seems to be the most performant. By a lot too, from my understanding (haven't ran any benchmark myself, but the readme shows some good postgres knowledge)

  • Sidekiq VS Que - a user suggested alternative
    2 projects | 3 Feb 2022
    Que seems like a good alternative if one doesn't want to use Reids. However, given that most apps need Redis (and have it within their infrastructure) nowadays, I still think that Sidekiq is the better option in the generic case.
  • Devious SQL: Message Queuing Using Native PostgreSQL
    7 projects | news.ycombinator.com | 28 Jan 2022
    Implementations that use advisory locks like https://github.com/que-rb/que are much more efficient (atleast when I last tested) and will easily reach 10k job/s on even very modest hardware.

    There is a Go port of Que but you can also easily port it to any language you like. I have a currently non-OSS implementation in Rust that I might OSS someday when I have time to clean it up.

  • Postgres is a great pub/sub and job server
    11 projects | news.ycombinator.com | 17 Dec 2021
    It’s also possible to use advisory locks to implement a job queue in Postgres. See e.g. Que[1]. Note there are a fair number of corner cases, so studying Que is wise if trying to implement something like this, as well as some (a bit older) elaboration[2].

    We implemented a similar design to Que for a specific use case in our application that has a known low volume of jobs and for a variety of reasons benefits from this design over other solutions.

    [1]: https://github.com/que-rb/que

  • Ruby Schedulers: Whenever vs Sidekiq Cron vs Sidekiq Scheduler
    2 projects | /r/ruby | 3 May 2021
    Do also take into consideration que-scheduler (disclaimer, am author). It is built on top of the robust que async job system.

oban

Posts with mentions or reviews of oban. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2024-04-16.
  • How to Use Flume in your Elixir Application
    2 projects | dev.to | 16 Apr 2024
    Oban, backed by PostgreSQL or SQLite, also provides a queue-based job processing system. Exq, on the other hand, is backed by Redis. It provides features similar to Flume, but without built-in rate limiting and batch processing capabilities.
  • Postgres as Queue
    8 projects | news.ycombinator.com | 9 Feb 2024
    In Elixir land Oban[0] uses Postgres as queue and seems to work quite well.

    [0] - https://github.com/sorentwo/oban

  • Zero Downtime Postgres Upgrades
    4 projects | news.ycombinator.com | 12 Dec 2023
    I hear you on that, and can say that Postgres is incredibly capable at going beyond typical relational database workloads. One example are durable queues that are transactionally consistent with the rest of the database play a unique role in our architecture that would otherwise require more ceremony. More details here: https://getoban.pro

    We are also working on shifting some workloads off of Postgres on to more appropriate systems as we scale, like logging. But we intentionally chose to minimize dependencies by pushing Postgres further to move faster, with migration plans ready as we continue to reach new levels of scale (e.g. using a dedicated log storage solution like elastic search or clickhouse).

  • Deno Cron
    15 projects | news.ycombinator.com | 29 Nov 2023
  • Switching to Elixir
    11 projects | news.ycombinator.com | 9 Nov 2023
    You can actually have "background jobs" in very different ways in Elixir.

    > I want background work to live on different compute capacity than http requests, both because they have very different resources usage

    In Elixir, because of the way the BEAM works (the unit of parallelism is much cheaper and consume a low amount of memory), "incoming http requests" and related "workers" are not as expensive (a lot less actually) compared to other stacks (for instance Ruby and Python), where it is quite critical to release "http workers" and not hold the connection (which is what lead to the creation of background job tools like Resque, DelayedJob, Sidekiq, Celery...).

    This means that you can actually hold incoming HTTP connections a lot longer without troubles.

    A consequence of this is that implementing "reverse proxies", or anything calling third party servers _right in the middle_ of your own HTTP call, is usually perfectly acceptable (something I've done more than a couple of times, the latest one powering the reverse proxy behind https://transport.data.gouv.fr - code available at https://github.com/etalab/transport-site/tree/master/apps/un...).

    As a consequence, what would be a bad pattern in Python or Ruby (holding the incoming HTTP connection) is not a problem with Elixir.

    > because I want to have state or queues in front of background work so there's a well-defined process for retry, error handling, and back-pressure.

    Unless you deal with immediate stuff like reverse proxying or cheap "one off async tasks" (like recording a metric), there also are solutions to have more "stateful" background works in Elixir, too.

    A popular background job queue is https://github.com/sorentwo/oban (roughly similar to Sidekiq at al), which uses Postgres.

    It handles retries, errors etc.

    But it's not the only solution, as you have other tools dedicated to processing, such as Broadway (https://github.com/dashbitco/broadway), which handles back-pressure, fault-tolerance, batching etc natively.

    You also have more simple options, such as flow (https://github.com/dashbitco/flow), gen_stage (https://github.com/elixir-lang/gen_stage), Task.async_stream (https://hexdocs.pm/elixir/1.12/Task.html#async_stream/5) etc.

    It allows to use the "right tool for the job" quite easily.

    It is also interesting to note there is no need to "go evented" if you need to fetch data from multiple HTTP servers: it can happen in the exact same process (even: in a background task attached to your HTTP server), as done here https://transport.data.gouv.fr/explore (if you zoom you will see vehicle moving in realtime, and ~80 data sources are being polled every 10 seconds & broadcasted to the visitors via pubsub & websockets).

  • Show HN: A simple API/CLI for scheduling HTTP requests
    2 projects | news.ycombinator.com | 27 Sep 2023
    Hi HN!

    This is something I've been tinkering on for the past couple months. It's basically just an API/CLI for scheduling delayed or recurring jobs as HTTP requests.

    I initially built it as a personal tool to save myself a bit of time on little side projects where I've needed scheduled/recurring alerts, but decided it could be a good opportunity to practice building out a nice landing page [0] and documentation [1]. And who knows, maybe someone else will find it useful Β―\_(ツ)_/Β―

    The tool relies heavily on Elixir's Oban [2] library for managing jobs, and Mintlify [3] for documentation. I also shamelessly stole most of the frontend design from Resend [4] because I'm a fan of the aesthetic and thought it would be good for my design chops to use their design as a guide. I also discovered Radix [5] UI while working on this, which ended up being immensely helpful for moving quickly on the frontend.

    Anyways, I almost certainly spent a bit too much time on small UX details that are most likely utterly inconsequential, but it was a fun exercise in polish :)

    All feedback is welcome!

    [0] https://www.booper.dev/

    [1] https://docs.booper.dev/

    [2] https://github.com/sorentwo/oban

    [3] https://mintlify.com/

    [4] https://resend.com/

    [5] https://www.radix-ui.com/

  • Choose Postgres Queue Technology
    17 projects | news.ycombinator.com | 24 Sep 2023
  • Pg_later: Asynchronous Queries for Postgres
    4 projects | news.ycombinator.com | 18 Aug 2023
    Idk about pgagent but any table is a resilient queue with the multiple locks available in pg along with some SELECT pg_advisory_lock or SELECT FOR UPDATE queries, and/or LISTEN/NOTIFY.

    Several bg job libs are built around native locking functionality

    > Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of schema.rb, and LISTEN/NOTIFY to reduce queuing latency.

    https://github.com/bensheldon/good_job

    > |> lock("FOR UPDATE SKIP LOCKED")

    https://github.com/sorentwo/oban/blob/8acfe4dcfb3e55bbf233aa...

  • Keep the Monolith, but Split the Workloads
    8 projects | news.ycombinator.com | 24 Apr 2023
    > Bad code in a specific part of the codebase bringing down the whole app, as in our November incident.

    This is a non-issue if you're using a Elixir/Erlang monolith given its fault tolerant nature.

    The noisy neighbour issue (resource hogging) is still something you need to manage though. If you use something like Oban[1] (for background job queues and cron jobs), you can set both local and global limits. Local being the current node, and global the cluster.

    Operating in a shared cluster (vs split workload deployments) give you the benefit of being much more efficient with your hardware. I've heard many stories of massive infra savings due to moving to an Elixir/Erlang system.

    1. https://github.com/sorentwo/oban

  • Library for reliably running jobs
    2 projects | /r/elixir | 23 Apr 2023

What are some alternatives?

When comparing Que and oban you can also consider the following projects:

Sidekiq - Simple, efficient background processing for Ruby

broadway - Concurrent and multi-stage data ingestion and data processing with Elixir

good_job - Multithreaded, Postgres-based, Active Job backend for Ruby on Rails.

exq - Job processing library for Elixir - compatible with Resque / Sidekiq

Delayed::Job - Database based asynchronous priority queue system -- Extracted from Shopify

Rihanna - Rihanna is a high performance postgres-backed job queue for Elixir

Resque - Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.

kafka_ex - Kafka client library for Elixir

Karafka - Ruby and Rails efficient multithreaded Kafka processing framework

verk - A job processing system that just verks! πŸ§›β€

Shoryuken - A super efficient Amazon SQS thread based message processor for Ruby

honeydew - Job Queue for Elixir. Clustered or Local. Straight BEAM. Optional Ecto. πŸ’ͺ🍈