rails_async_methods VS solid_queue

You can check it out here: https://github.com/benngarcia/rails_async_methods

One of the key challenges in any applications is managing long-running tasks without affecting the user experience. PopaDex leverages the solid_queue gem to handle background processing efficiently. This "DB-based queuing backend for Active Job" allows for tasks such as report generation and notifications to be processed in the background, ensuring the application remains responsive. The beauty of solid_queue lies in its simplicity and efficiency, obviating the need for more complex solutions like Redis or Sidekiq for background job management. This choice offers several distinct advantages:

Once we are done with default gems, should we look into something we usually use? That's jwt because we need session tokens for our API. Next comes our one and only sidekiq. For a long period of time it was the best in town solution for background jobs. Now we could also consider solid_queue or good_job. In development and testing groups we need rspec-rails, factory_bot_rails and ffaker. Dealing with money? Start doing it properly from the beginning! Do not forget to install money-rails. Once everything is added to the Gemfile do not forget to trigger bundle install.

Rails isn't super opinionated about database writes, its mostly left up to developers to discover that for relational DBs you do not want to be doing a bunch of small writes all at once.

That said it specifically has tools to address this that started appearing a few years ago https://github.com/rails/rails/pull/35077

The way my team handles it is to stick Kafka in between whats generating the records (for us, a bunch of web scraping workers) and and a consumer that pulls off the Kafka queue and runs an insert when its internal buffer reaches around 50k rows.

Rails is also looking to add some more direct background type work with https://github.com/basecamp/solid_queue but this is still very new - most larger Rails shops are going to be running a second system and a gem called Sidekiq that pulls jobs out of Redis.