heroku-buildpack-jemalloc VS Delayed::Job

Using jemalloc instead of regular malloc helps too. The exact way to do this depends on the platform you use, but it is pretty simple on Heroku. Just set heroku-buildpack-jemalloc as the first buildpack (ahead of the heroku/ruby buildpack).

I've recently discovered jemalloc, specifically when used with Heroku.

"Using jemalloc instead of regular malloc helps too. The exact way to do this depends on the platform you use, but it is pretty simple on Heroku. Just set heroku-buildpack-jemalloc as the first buildpack (ahead of the heroku/ruby buildpack)."

FYI, remember to set JEMALLOC_ENABLED=true in your env to actually turn it on.

https://github.com/gaffneyc/heroku-buildpack-jemalloc

Like I mentioned earlier, Digital Ocean App Platform uses the same buildpacks as Heroku to deploy your apps. This means that most apps that can be deployed on Heroku should also be deployed on Digital Ocean. There is one big caveat, though; you can't select which buildpack(s) to use. This means you have to rely on Digital Ocean to pick the right ones for your project. It also gives you a bit less flexibility in how your app runs. For instance, I recently configured our app at work to run using jemalloc, a malloc alternative that often has better performance for Ruby apps. We did that via a buildpack heroku-buildpack-jemalloc, which allowed us to switch to jemalloc without any app changes. Customizations to the build environment like this don't seem possible given the Digital Ocean App Platform's current offerings.

I included this in the github issue but we had coverband induced log spam, it got fixed and suddenly we hit the bug. I did find this from someone with a similar issue but there was no conclusion.

So how do we trigger such a long-running process from a Rails request? The first option that comes to mind is a background job run by some of the queuing back-ends such as Sidekiq, Resque or DelayedJob, possibly governed by ActiveJob. While this would surely work, the problem with all these solutions is that they usually have a limited number of workers available on the server and we didn’t want to potentially block other important background tasks for so long.

Several gems support job queues and background processing in the Rails world — Delayed Job and Sidekiq being the two most popular ones.

Rails 7 support only just got added https://github.com/collectiveidea/delayed_job/pull/1161

Back in the day, before Sidekiq and such, we used Delayed Job https://github.com/collectiveidea/delayed_job

There are a few of popular systems. A few need a database, such as Delayed::Job, while others prefer Redis, such as Resque and Sidekiq.

I always wad a fan of Sidekiq, but recently I wanted to remove redis as a dependency. I ended up going back to classic delayed_job. My reason for that was I also didn't want dependency with Postgres, as I like to use sqlite on development and Postgres on production. The best solution I've seen is Que, which works with Postgres. It is complete, has all the features you need.

Ruby gets hammered a lot for its green threads and no real concurrency yet Ruby libraries such as Resque, Delayed Job and Sidekiq are some of the most popular choices to run background jobs in the industry. When you have a huge project and millions of requests and hundreds of thousands of operations to perform, reach out to one of these solutions which are feature complete and you don’t have to reinvent the wheel as its complex piece of software to implement and requires thousands of human hours. While they help you scale sometimes they are not what you are looking for or need as they are meant for a scale of 10000s of jobs per second and what you need is a background processing for few jobs that can be handled in memory without a dependency of a queue such as Redis. If you look at the benchmark numbers from Sidekiq they are for 100K jobs because that is the scale it is meant to be though one can use it for running a few hundred complex jobs as well where you require queue management and supervision.

Before we dive in, I wanted to go over my background real quick for context. I'm mostly a Rails developer professionally, and I am on the 'Infrastructure' team at work, so I integrate closely with Heroku and am pretty familiar with their product and offerings. Given my Rails background, this article will focus on deploying a web app with at least one background job process. For my hackathon app, I had a web process running Puma and a worker process running DelayedJob.