Getting Sidekiq to play nicely with auto-scaling

• Sidekiq

  91 12,950 8.9 Ruby

  Simple, efficient background processing for Ruby

  

Has anyone had trouble with Sidekiq jobs disappearing when servers shutdown? We're on Heroku and when we tried to use Rails Autoscale, we would frequently see jobs just evaporate. The logs for a job would stop mid-processing. Sidekiq would not push the job back onto the queue and retry it. We upgraded to Enterprise and tried out SuperFetch, but then found that instead jobs were frequently getting duplicated. The Sidekiq docs mention support for rolling restarts, but it doesn't appear that works on Heroku. It's pretty important for us that jobs run once and only once, but I'm not sure how to accomplish that without using a database backed queuing system for locking.

• job-iteration

  5 1,025 8.6 Ruby

  Makes your background jobs interruptible and resumable by design.

  

If that's the case, https://github.com/Shopify/job-iteration could help by making all your jobs easily interruptible.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• sidekiq-unique-jobs

  4 1,419 7.9 Ruby

  Prevents duplicate Sidekiq jobs

That's an interesting suggestion. We're relying on ActiveJob and sidekiq-unique-jobs doesn't explicitly support it, unfortunately. We'll have to test it out, though, and see if it just happens to work.

Suggest a related project