Sidekiq VS Resque

I like Solid Queue and the direction things are heading, but its hard to overlook the performance. A system that does tens to hundreds of thousands of jobs/s on Sidekiq + Redis, will now get bottlenecked by transactional performance with solid queue / PG - https://github.com/sidekiq/sidekiq/wiki/Active-Job#performan...

# copied from https://github.com/sidekiq/sidekiq/blob/main/Gemfile source "https://rubygems.org" gemspec gem "rake" RAILS_VERSION = "~> 8.0" gem "actionmailer", RAILS_VERSION gem "actionpack", RAILS_VERSION gem "activejob", RAILS_VERSION gem "activerecord", RAILS_VERSION gem "railties", RAILS_VERSION gem "redis-client" # gem "bumbler" # gem "debug" gem "sqlite3", "~> 2.2", platforms: :ruby gem "activerecord-jdbcsqlite3-adapter", platforms: :jruby gem "after_commit_everywhere", require: false gem "yard" gem "csv" gem "vernier" unless RUBY_VERSION < "3" gem "webrick" group :test do gem "maxitest" gem "simplecov" gem "debug" end group :development, :test do gem "standard", require: false end group :load_test do gem "toxiproxy" gem "ruby-prof" end

If you’re developing in Node, BullMQ has been rising in popularity as a go-to solution. For Rails applications, you can use ActiveJob with backends like Sidekiq for efficient background job processing.

Sidekiq is known for its fast and efficient processing using threads in Ruby, which allows it to handle many jobs concurrently.

I'm not sure I personally agree with this, and I'm not 100% sure the developer community at-large does either...

Let's take a few examples, which I've shared elsewhere in similar discussions:

- GitLab: Open Source or Open Core? Most would say Open Source, but (I assume) you would argue Open Core [0].

- Plausible: Open Source or Open Core? They say Open Source, but it's actually Open Core [1].

- Cal.com: Open Source or Open Core? They say Open Source, but once again, Open Core [2].

- Posthog: Open Source or Open Core? They say Open Source, still Open Core [3].

- Sidekiq: Open Source or Open Core? Once again: Open Core [4].

Yet, every dev I know would consider these projects Open Source. So there's a disconnect somewhere.

Under this mindset, very few open source startups are actually open source, yet everybody says they are?

I'm not trying to argue either way; I'm trying to point out a disconnect here.

[0]: https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/LICENS...

[1]: https://github.com/plausible/analytics/blob/2dd2f058d1dcae6f...

[2]: https://github.com/calcom/cal.com/blob/main/packages/feature...

[3]: https://github.com/PostHog/posthog/blob/master/ee/LICENSE

[4]: https://github.com/sidekiq/sidekiq/blob/main/COMM-LICENSE.tx...

Okay, back to our Rails app. In our app, the lifetime should be the whole Sidekiq process. Luckily, Sidekiq has internal documentation on how it runs. I won't copy the entire documentation here, just the part we are interested in:

Rollbar is a great error-tracking service. It alerts us on exceptions and errors, provides analysis tools and dashboard, so we can see, reproduce, and fix bugs quickly when something went wrong. This service has a possibility to log not only uncaught exceptions but any messages. By default, the messages are reported synchronously, but you can enable asynchronous reporting using Sidekiq, girl_friday, or Resque. Also, you can provide your own handler and a failover handler to be confident, that your error is tracked and delivered in the case of primary handler’s fail.

Hi there! I want to show off a little feature I made using hanami, htmx and a little bit of redis + sidekiq.

I'd say Sidekiq is the top competitor here.

There's something wrong at Redislabs, it took them over a year to get RESP3 rolled out into their hosted service, you'd expect a rollout of that to be a bit quicker when they're the owner of Redis.

It affected us when upgrading Sidekiq to version 7, which dropped support for older Redis, and their Envoy proxy setup didn't support HELLO and RESP3: https://github.com/sidekiq/sidekiq/issues/5594

Resque relies on Redis for job queue management and is known for its scalability and efficiency.

We split the staging server into two instances using Ubuntu 22.04 as the base image. The first instance is for a web server with nginx,  passenger and MySQL. The second instance is for the support server and this is where we install redis, memcache, mongodb and resque.

It is hard to imagine any big and complex Rails project without background jobs processing. There are many gems for this task: **Delayed Job, Sidekiq, Resque, SuckerPunch** and more. And Active Job has arrived here to rule them all.

Rollbar is a great error-tracking service. It alerts us on exceptions and errors, provides analysis tools and dashboard, so we can see, reproduce, and fix bugs quickly when something went wrong. This service has a possibility to log not only uncaught exceptions but any messages. By default, the messages are reported synchronously, but you can enable asynchronous reporting using Sidekiq, girl_friday, or Resque. Also, you can provide your own handler and a failover handler to be confident, that your error is tracked and delivered in the case of primary handler’s fail.

The free version acts exactly like Resque, the previous market leader in Ruby background jobs. If it was good enough reliability for GitHub and Shopify to use for years, it was good enough for Sidekiq OSS too.

Here's Resque literally using `lpop` which is destructive and will lose jobs.

https://github.com/resque/resque/blob/7623b8dfbdd0a07eb04b19...

You can use a background job queue like Resque to scrape and process data in the background, and a scheduler like resque-scheduler to schedule jobs to run your scraper periodically.

So how do we trigger such a long-running process from a Rails request? The first option that comes to mind is a background job run by some of the queuing back-ends such as Sidekiq, Resque or DelayedJob, possibly governed by ActiveJob. While this would surely work, the problem with all these solutions is that they usually have a limited number of workers available on the server and we didn’t want to potentially block other important background tasks for so long.

Background jobs are another limitation. Since only the Aha! web service runs in a dynamic staging, the host environment's workers would process any Resque jobs that were sent to the shared Redis instance. If your branch hadn't updated any background-able methods, this would be no big deal. But if you were hoping to test changes to these methods, you would be out of luck.

Redis Lists are quite versatile and used as the backbone for implementing scalable architectural patterns such as consumer-producer (based on queues), where producer applications push items into a List, and consumers (also called workers) process those items. Popular projects such as resque, sidekiq, celery etc. use Redis behind the scenes to implement background jobs.