Que VS Delayed::Job

> Can you define "low throughput"?

<1000 messages per minute

Not saying SKIP LOCKED can't work with that many. But you'll probably want to do something better.

FWIW, Que uses advisory locks [1]

[1] https://github.com/que-rb/que

Probably worth mentioning that aside from delayed_job there are at least two more modern alternatives backed by the DB: Que and good_job.

Good article. Sidekiq is a good, well respected too. However if you are starting out I would recommend not using it, and instead choosing a DB based queue system. We have great success with que, but there are others like good_job.

(not sure why this comment was dead, I vouched for it)

There are a lot of ways to implement a queue in an RDBMS and a lot of those ways are naive to locking behavior. That said, with PostgreSQL specifically, there are some techniques that result in an efficient queue without locking problems. The article doesn't really talk about their implementation so we can't know what they did, but one open source example is Que[1]. Que uses a combination of advisory locking rather than row-level locks and notification channels to great effect, as you can read in the README.

[1]: https://github.com/que-rb/que

https://github.com/que-rb/que

This one seems to be the most performant. By a lot too, from my understanding (haven't ran any benchmark myself, but the readme shows some good postgres knowledge)

Que seems like a good alternative if one doesn't want to use Reids. However, given that most apps need Redis (and have it within their infrastructure) nowadays, I still think that Sidekiq is the better option in the generic case.

Implementations that use advisory locks like https://github.com/que-rb/que are much more efficient (atleast when I last tested) and will easily reach 10k job/s on even very modest hardware.

There is a Go port of Que but you can also easily port it to any language you like. I have a currently non-OSS implementation in Rust that I might OSS someday when I have time to clean it up.

It’s also possible to use advisory locks to implement a job queue in Postgres. See e.g. Que[1]. Note there are a fair number of corner cases, so studying Que is wise if trying to implement something like this, as well as some (a bit older) elaboration[2].

We implemented a similar design to Que for a specific use case in our application that has a known low volume of jobs and for a variety of reasons benefits from this design over other solutions.

[1]: https://github.com/que-rb/que

Do also take into consideration que-scheduler (disclaimer, am author). It is built on top of the robust que async job system.

I included this in the github issue but we had coverband induced log spam, it got fixed and suddenly we hit the bug. I did find this from someone with a similar issue but there was no conclusion.

So how do we trigger such a long-running process from a Rails request? The first option that comes to mind is a background job run by some of the queuing back-ends such as Sidekiq, Resque or DelayedJob, possibly governed by ActiveJob. While this would surely work, the problem with all these solutions is that they usually have a limited number of workers available on the server and we didn’t want to potentially block other important background tasks for so long.

Several gems support job queues and background processing in the Rails world — Delayed Job and Sidekiq being the two most popular ones.

Rails 7 support only just got added https://github.com/collectiveidea/delayed_job/pull/1161

Back in the day, before Sidekiq and such, we used Delayed Job https://github.com/collectiveidea/delayed_job

There are a few of popular systems. A few need a database, such as Delayed::Job, while others prefer Redis, such as Resque and Sidekiq.

I always wad a fan of Sidekiq, but recently I wanted to remove redis as a dependency. I ended up going back to classic delayed_job. My reason for that was I also didn't want dependency with Postgres, as I like to use sqlite on development and Postgres on production. The best solution I've seen is Que, which works with Postgres. It is complete, has all the features you need.

Ruby gets hammered a lot for its green threads and no real concurrency yet Ruby libraries such as Resque, Delayed Job and Sidekiq are some of the most popular choices to run background jobs in the industry. When you have a huge project and millions of requests and hundreds of thousands of operations to perform, reach out to one of these solutions which are feature complete and you don’t have to reinvent the wheel as its complex piece of software to implement and requires thousands of human hours. While they help you scale sometimes they are not what you are looking for or need as they are meant for a scale of 10000s of jobs per second and what you need is a background processing for few jobs that can be handled in memory without a dependency of a queue such as Redis. If you look at the benchmark numbers from Sidekiq they are for 100K jobs because that is the scale it is meant to be though one can use it for running a few hundred complex jobs as well where you require queue management and supervision.

Before we dive in, I wanted to go over my background real quick for context. I'm mostly a Rails developer professionally, and I am on the 'Infrastructure' team at work, so I integrate closely with Heroku and am pretty familiar with their product and offerings. Given my Rails background, this article will focus on deploying a web app with at least one background job process. For my hackathon app, I had a web process running Puma and a worker process running DelayedJob.