Sidekiq
Delayed::Job
Our great sponsors
Sidekiq | Delayed::Job | |
---|---|---|
91 | 10 | |
12,931 | 4,801 | |
0.4% | 0.1% | |
8.9 | 2.1 | |
9 days ago | 2 months ago | |
Ruby | Ruby | |
GNU Lesser General Public License v3.0 only | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Sidekiq
-
solid_queue alternatives - Sidekiq and good_job
3 projects | 21 Apr 2024
I'd say Sidekiq is the top competitor here.
-
Valkey Is Rapidly Overtaking Redis
There's something wrong at Redislabs, it took them over a year to get RESP3 rolled out into their hosted service, you'd expect a rollout of that to be a bit quicker when they're the owner of Redis.
It affected us when upgrading Sidekiq to version 7, which dropped support for older Redis, and their Envoy proxy setup didn't support HELLO and RESP3: https://github.com/sidekiq/sidekiq/issues/5594
-
Redis Re-Implemented with SQLite
That depends on how the `maxmemory-policy` is configured, and queue systems based on Redis will tell you not to allow eviction. https://github.com/sidekiq/sidekiq/wiki/Using-Redis#memory (it even logs a warnings if it detects your Redis is misconfigured IIRC).
-
3 one-person million dollar online businesses
Sidekiq https://sidekiq.org/: This one started as an open source project, once it got enough traction, the developer made a premium version of it, and makes money by selling licenses to businesses.
-
Choose Postgres Queue Technology
Sidekiq will drop in-progress jobs when a worker crashes. Sidekiq Pro can recover those jobs but with a large delay. Sidekiq is excellent overall but it’s not suitable for processing critical jobs with a low latency guarantee.
https://github.com/sidekiq/sidekiq/wiki/Reliability
-
We built the fastest CI in the world. It failed
> I'm not sure feature withholding has traditionally worked out well in the developer space.
I think it's worked out well for Sidekiq (https://sidekiq.org). I really like their model of layering valuable features between the OSS / Pro / Enterprise licenses.
-
Exploring concurrent rate limiters, mutexes, semaphores
I was studying Sidekiq's page on rate limiters. The first type of rate limiting mentioned is the concurrent limiter: only n tasks are allowed to run at any point in time. Note that this is independent of time units (e.g. per second), or how long they take to run. The only limitation is the number of concurrent tasks/requests.
- Ask HN: What are some of the most elegant codebases in your favorite language?
- Sidekiq and managing resumable jobs?
-
Organize Business Logic in Your Ruby on Rails Application
The code above isn't idempotent. If you run it twice, it will create two copies, which is probably not what you intended. Why is this important? Because most backend job processors like Sidekiq don't make any guarantees that your jobs will run exactly once.
Delayed::Job
-
How to narrow down race condition in delayed job/mysql2?
I included this in the github issue but we had coverband induced log spam, it got fixed and suddenly we hit the bug. I did find this from someone with a similar issue but there was no conclusion.
-
How to run a really long task from a Rails web request
So how do we trigger such a long-running process from a Rails request? The first option that comes to mind is a background job run by some of the queuing back-ends such as Sidekiq, Resque or DelayedJob, possibly governed by ActiveJob. While this would surely work, the problem with all these solutions is that they usually have a limited number of workers available on the server and we didn’t want to potentially block other important background tasks for so long.
-
Delayed Job vs. Sidekiq: Which Is Better?
Several gems support job queues and background processing in the Rails world — Delayed Job and Sidekiq being the two most popular ones.
Rails 7 support only just got added https://github.com/collectiveidea/delayed_job/pull/1161
-
Why does rails have a tradition of queuing background jobs in a separate NoSQL store, when both the queueing controller and the job class tend to hammer the main database anyway?
Back in the day, before Sidekiq and such, we used Delayed Job https://github.com/collectiveidea/delayed_job
-
A quick look at background jobs in Ruby
There are a few of popular systems. A few need a database, such as Delayed::Job, while others prefer Redis, such as Resque and Sidekiq.
-
Schedule background jobs in Rails with Integromat
I always wad a fan of Sidekiq, but recently I wanted to remove redis as a dependency. I ended up going back to classic delayed_job. My reason for that was I also didn't want dependency with Postgres, as I like to use sqlite on development and Postgres on production. The best solution I've seen is Que, which works with Postgres. It is complete, has all the features you need.
-
Background Job Processing in Ruby without external libraries and dependencies
Ruby gets hammered a lot for its green threads and no real concurrency yet Ruby libraries such as Resque, Delayed Job and Sidekiq are some of the most popular choices to run background jobs in the industry. When you have a huge project and millions of requests and hundreds of thousands of operations to perform, reach out to one of these solutions which are feature complete and you don’t have to reinvent the wheel as its complex piece of software to implement and requires thousands of human hours. While they help you scale sometimes they are not what you are looking for or need as they are meant for a scale of 10000s of jobs per second and what you need is a background processing for few jobs that can be handled in memory without a dependency of a queue such as Redis. If you look at the benchmark numbers from Sidekiq they are for 100K jobs because that is the scale it is meant to be though one can use it for running a few hundred complex jobs as well where you require queue management and supervision.
-
Digital Ocean App Platform vs Heroku
Before we dive in, I wanted to go over my background real quick for context. I'm mostly a Rails developer professionally, and I am on the 'Infrastructure' team at work, so I integrate closely with Heroku and am pretty familiar with their product and offerings. Given my Rails background, this article will focus on deploying a web app with at least one background job process. For my hackathon app, I had a web process running Puma and a worker process running DelayedJob.
What are some alternatives?
Resque - Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.
Sucker Punch - Sucker Punch is a Ruby asynchronous processing library using concurrent-ruby, heavily influenced by Sidekiq and girl_friday.
Sneakers - A fast background processing framework for Ruby and RabbitMQ
good_job - Multithreaded, Postgres-based, Active Job backend for Ruby on Rails.
Shoryuken - A super efficient Amazon SQS thread based message processor for Ruby
Que - A Ruby job queue that uses PostgreSQL's advisory locks for speed and reliability.
Apache Kafka - Mirror of Apache Kafka
celery - Distributed Task Queue (development branch)
Karafka - Ruby and Rails efficient multithreaded Kafka processing framework