SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue

1. tembo

   1 5 1,231 7.7 TypeScript

   one cli, all the coding agents

   

   Love to see it. We (CoreDB) recently released PGMQ, a message queue extension for Postgres: https://github.com/CoreDB-io/coredb/extensions/pgmq

2. Sevalla

   sevalla.com featured

   Deploy and host your apps and databases, now with $50 credit! Sevalla is the PaaS you have been looking for! Advanced deployment pipelines, usage-based pricing, preview apps, templates, human support by developers, and much more!

   

3. Que

   2 11 2,311 5.8 Ruby

   A Ruby job queue that uses PostgreSQL's advisory locks for speed and reliability.

   

   (not sure why this comment was dead, I vouched for it)

   There are a lot of ways to implement a queue in an RDBMS and a lot of those ways are naive to locking behavior. That said, with PostgreSQL specifically, there are some techniques that result in an efficient queue without locking problems. The article doesn't really talk about their implementation so we can't know what they did, but one open source example is Que[1]. Que uses a combination of advisory locking rather than row-level locks and notification channels to great effect, as you can read in the README.

   [1]: https://github.com/que-rb/que

4. pg-boss

   3 24 2,667 7.0 JavaScript

   Queueing jobs in Postgres from Node.js like a boss

   

   If you don't want to roll your own, look into https://github.com/timgit/pg-boss

5. litequeue

   4 3 208 4.7 Python

   Queue built on top of SQLite

   

   SQLite is missing some features like `SELECT FOR UPDATE`, but you can work around some issues with a few extra queries. I wrote litequeue[0] with this specific purpose. I haven't been able to use it a lot, so I don't have real-world numbers of how it scales, but the scaling limits depend on how fast you can insert into the database.

   [0]: https://github.com/litements/litequeue

6. worker

   5 23 2,011 9.1 TypeScript

   High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)

   

   Another good library for this is Graphile Worker:

   https://github.com/graphile/worker

   Uses both listen notify and advisory locks so it is using all the right features. And you can enqueue a job from sql and plpgsql triggers. Nice!

   Worker is in Node js.

   https://github.com/graphile/worker

7. good_job

   6 40 2,859 8.5 Ruby

   Multithreaded, Postgres-based, Active Job backend for Ruby on Rails.

   

   I'm the GoodJob author. Here's the class that is responsible for implementing Postgres's LISTEN/NOTIFY functionality in GoodJob:

   https://github.com/bensheldon/good_job/blob/10e9d9b714a668dc...

   That's heavily inspired by Rail's Action Cable (websockets) Adapter for Postgres, which is a bit simpler and easier to understand:

   https://github.com/rails/rails/blob/be287ac0d5000e667510faba...

   Briefly, it spins up a background thread with a dedicated database connection and doings a blocking Postgres LISTEN query returns results, and then it forwards the result to other subscribing objects.

8. Ruby on Rails

   7 544 57,364 10.0 Ruby

   Ruby on Rails

   

   I'm the GoodJob author. Here's the class that is responsible for implementing Postgres's LISTEN/NOTIFY functionality in GoodJob:

   https://github.com/bensheldon/good_job/blob/10e9d9b714a668dc...

   That's heavily inspired by Rail's Action Cable (websockets) Adapter for Postgres, which is a bit simpler and easier to understand:

   https://github.com/rails/rails/blob/be287ac0d5000e667510faba...

   Briefly, it spins up a background thread with a dedicated database connection and doings a blocking Postgres LISTEN query returns results, and then it forwards the result to other subscribing objects.

9. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

10. starqueue

    8 4 0 10.0 Python

    I wrote a message queue in Python called StarQueue.

    It’s meant to be a simpler reimagining of Amazon SQS.

    It has an HTTP API and behaves mostly like SQS.

    I wrote it to support Postgres, Microsoft’s SQL server and so MySQL because they all support SKIP LOCKED.

    At some point I turned it into a hosted service and only maintained the Postgres implementation though the MySQL and SQL server code is still in there.

    It’s not an active project but the code is at https://github.com/starqueue/starqueue/

11. tqs

    9 1 6 10.0 Python

    Tiny Queue Service (Server)

    

    I wrote https://github.com/TinyWebServices/tqs a couple of years ago. It is modelled after SQS and runs in a single threaded Tornado server.

    I don’t know how many messages per second it does but for a podcast crawling side project I have processed billions of messages through this little Python wrapper around SQLite. Zero problems. It just keeps running happily.

12. amqp

    10 1 689 6.8 Elixir

    Idiomatic Elixir client for RabbitMQ (by pma)

    

    So far I have not had any issues with the Elixir one.

    https://github.com/pma/amqp

13. pgtt

    11 1 91 7.0 C

    PostgreSQL extension to create, manage and use Oracle-style Global Temporary Tables and the others RDBMS (by darold)

    

14. oban

    12 33 3,627 9.2 Elixir

    💎 Robust job processing in Elixir, backed by modern PostgreSQL, SQLite3, and MySQL

    

    Oban [0] is a job queuing system in the Elixir world that supports both postgres and sqlite.

    [0] https://getoban.pro/

15. rq

    13 30 10,320 9.0 Python

    Simple job queues for Python

    

    Also had a similar experience using RabbitMQ with Django+Celery. Extremely complicated and workers/queues would just stop for no reason.

    Moved to Python-RQ [1] + Redis and been rock solid for years now.

    [1] https://python-rq.org/

16. neoq

    14 7 320 6.4 Go

    Queue-agnostic background job library for Go, with a pleasant API and powerful features.

    

    This is exactly the thesis behind neoq: https://github.com/acaloiaro/neoq

17. BeanstalkD

    15 15 6,646 5.0 C

    Beanstalk is a simple, fast work queue.

    

    Not when a queue is involved. IME trying to replicate something like beanstalkd (https://beanstalkd.github.io/) in postgres is asking for trouble for anything but trivial workloads.

    If you're measuring throughput in jobs/s, use a real work queue.

18. SaaSHub

    www.saashub.com featured

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    

Suggest a related project