Choose Postgres Queue Technology

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • oban

    💎 Robust job processing in Elixir, backed by modern PostgreSQL and SQLite3

    This is exactly what the Oban https://getoban.pro/ Elixir library uses and combining postgres plus actors for queues scales pretty great for 90% of the needs out there. I have used it at my last few jobs at pretty decent scale and would take it over 10 years using Celery to manage queues + supervisord, setting up RabbitMQ or Redis. Its so simple you only need Elixir and Postgres and not 3 or 4 infrastructure pieces to manage a queue.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
  • Que

    A Ruby job queue that uses PostgreSQL's advisory locks for speed and reliability.

    Few things.

    1. The main downside to using PostgreSQL as a pub/sub bus with LISTEN/NOTIFY is that LISTEN is a session feature, making it incompatible with statement level connection pooling.

    2. If you are going to do this use advisory locks [0]. Other forms of explicit locking put more pressure on the database while advisory locks are deliberately very lightweight.

    My favorite example implementation is que [1] which is ported to several languages.

    [0] https://www.postgresql.org/docs/current/explicit-locking.htm...

    [1] https://github.com/que-rb/que

  • pg-boss

    Queueing jobs in Postgres from Node.js like a boss

    For running queues on Postgres with Node.js backend(s), I highly recommend https://github.com/timgit/pg-boss. I'm sure it has it scale limits. But if you're one of the 90% of the apps that never needs any kind of scale that a modern server can't easily handle then it's fantastic. You get transactional queueing of jobs, and it automatically handles syncing across multiple job processing servers using Postgres locks.

  • worker

    High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)

    I do enjoy using https://github.com/graphile/worker for my postgresql queuing needs. Very scalable, the next release 0.14 even more so, and easy to use.

  • good_job

    Multithreaded, Postgres-based, Active Job backend for Ruby on Rails.

    For Rails apps, you can do this using the ActiveJob interface via

    https://github.com/bensheldon/good_job

    Had it in production for about a quarter and it’s worked well.

  • starqueue

    MS SQL server, Postgres and MySQL all support SKIP LOCKED, which means they are all suitable for running queues.

    I built a complete implementation in Python designed to work the same as SQS but be more simple:

    https://github.com/starqueue/starqueue

    Alternatively if you just want to quickly hack something into your application, here is a complete solution in Python with retries:

        import psycopg2

  • arniesmtpbufferserver

    My guess is that many people are implementing queuing mechanisms just for sending email.

    The Linux file system makes a perfectly good basis for a message queue since file moves are atomic.

    You can see how this works in Arnie SMTP buffer server, a super simple queue just for emails, no database at all, just the file system.

    https://github.com/bootrino/arniesmtpbufferserver

  • windmill

    Open-source developer platform to power your entire infra and turn scripts into webhooks, workflows and UIs. Fastest workflow engine (13x vs Airflow). Open-source alternative to Retool and Temporal.

    We use exactly this for windmill (OSS Retool alternative + modern airflow) and run benchmarks everyday. On a modest github CI instance with our workers and postgres run as container, our benchmarks run at 1200jobs/s. Workers can be added and it will scale gracefully up to 5000jobs/s. We are exploring using Citus to cross the barrier of 5000j/s on our multi-tenant instance.

    https://github.com/windmill-labs/windmill/tree/benchmarks

  • kubeblocks

    KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.

    https://github.com/apecloud/kubeblocks/blob/main/docs/releas...

  • litequeue

    Queue built on top of SQLite

    To make sure you that the message you are trying to retrieve hasn't been locked already by another worker.

    [0]: https://github.com/litements/litequeue/

    [1]: https://github.com/litements/litequeue/blob/3fece7aa9e9a31e4...

  • neoq

    Queue-agnostic background job library for Go, with a pleasant API and powerful features.

    I just want to commend OP - if they’re here - for choosing an int64 for job IDs, and MD5 for hashing the payload in Neoq, the job library linked [0] from the article.

    Especially given the emphasis on YAGNI, you don’t need a UUID primary key, and all of its problems they bring for B+trees (that thing RDBMS is built on), nor do you need the collision resistance of SHA256 - the odds of you creating a dupe job hash with MD5 are vanishingly small.

    As to the actual topic, it’s fine IFF you carefully monitor for accumulating dead tuples, and adjust auto-vacuum for that table as necessary. While not something you’d run into at the start, at a modest scale you may start to see issues. May. You may also opt to switch to Redis or something else before that point anyway.

    [0]: https://github.com/acaloiaro/neoq

  • pgjobq

    Atomic low latency job queues running on Postgres

    I feel like one of the problems with using Postgres as a queue is that it’s hard to get started. There’s a lot you need to know. Getting started with something like Pub/Sun on GCP is much easier for many developers.

    I’ve experimented with making this easier via libraries that provide high-level APIs for using Postgres as a queue and manage the schemas, listen/notify, etc for you: https://github.com/adriangb/pgjobq

  • pgmq

    A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.

    PGMQ does not require a client library, https://github.com/tembo-io/pgmq so long as your language of choice can run SQL. All the functions live in Postgres, and you just call them with SQL statement. Very similar feel and semantics to SQS.

  • Sidekiq

    Simple, efficient background processing for Ruby

    Sidekiq will drop in-progress jobs when a worker crashes. Sidekiq Pro can recover those jobs but with a large delay. Sidekiq is excellent overall but it’s not suitable for processing critical jobs with a low latency guarantee.

    https://github.com/sidekiq/sidekiq/wiki/Reliability

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Pg-boss: Queueing jobs in Postgres from Node.js

    1 project | news.ycombinator.com | 27 Aug 2024
  • Show HN: PgQueuer – Transform Your PostgreSQL into a Powerful Job Queue

    23 projects | news.ycombinator.com | 18 Aug 2024
  • Show HN: An SQS Alternative on Postgres

    10 projects | news.ycombinator.com | 9 May 2024
  • Wasp x Supabase: Smokin’ Hot Full-Stack Combo 🌶️ 🔥

    8 projects | dev.to | 8 May 2024
  • Minimal downtime major PostgreSQL version upgrades with pg_easy_replicate

    1 project | /r/PostgreSQL | 27 Jun 2023