Devious SQL: Message Queuing Using Native PostgreSQL

• django-postgres-queue

  4 116 0.0 Python

  A task queue for django

• pg-boss

  12 1,615 5.5 JavaScript

  Queueing jobs in Node.js using PostgreSQL like a boss

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Que

  10 2,284 6.0 Ruby

  A Ruby job queue that uses PostgreSQL's advisory locks for speed and reliability.

  

Implementations that use advisory locks like https://github.com/que-rb/que are much more efficient (atleast when I last tested) and will easily reach 10k job/s on even very modest hardware.

There is a Go port of Que but you can also easily port it to any language you like. I have a currently non-OSS implementation in Rust that I might OSS someday when I have time to clean it up.

• worker

  19 1,633 9.7 TypeScript

  High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)

  

We're currently keeping it as simple as possible. We just use a table and this query that runs every second: `DELETE FROM job_queue RETURNING type, info;`. Works great and reliably so far. I think our current volume is on the order of 100-1000 req/sec.

We'll likely switch to graphile-worker when we need more performance, but we're all about avoiding premature optimization. That library has been benchmarked to handle 10k req/sec: https://github.com/graphile/worker#performance

• dramatiq-pg

  4 - -

• redislite

  1 564 5.0 Python

  Redis in a python module.

  

If you are developing on Linux, rq + redis is probably the simplest message queueing system around. Redis can even be installed from pip,

https://pypi.org/project/redis/

or run within python,

https://github.com/yahoo/redislite

it's pretty much the easiest thing to deploy.

• OddJob

  1 16 0.0 C#

  OddJob - A Simple Job Scheduler for .NET

Probably not the one you're thinking of, but I wrote something like this for C# once upon a time called OddJob [0]. If nothing else it shows decent patterns for a job queue that is friendly towards multiple producers/consumers.

[0] - https://github.com/to11mtm/oddjob/tree/cleaning-aug-19

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project