resque-scheduler VS Whenever

You can use a background job queue like Resque to scrape and process data in the background, and a scheduler like resque-scheduler to schedule jobs to run your scraper periodically.

Learn about Unix cron and use the whenever gem.

there are cronjob gems available, i believe https://github.com/javan/whenever or so seems to be used frequently

- Use them on a specific schedule as needed by your application, such as taking backups, sending periodic reminders or notifications for users, cleaning the database tables, etc. You can use a gem such as Whenever for managing scheduled cron jobs.

whenever gem is used in Rails applications to schedule cron jobs e.g. send email notification about monthly expenditure on the 1st of each month.

Then you can use the whenever gem to schedule stuff.

- [whenever](https://github.com/javan/whenever)

whenever

There are multiple ways to achieve this. Probably the simplest and most reliable is to rely on cron to run your process every minute. Cron keeps track of time and executes your script each minute. This has the added benefit of avoiding a long running process (which can have its own issues). You can even have your script install it’s own cron job using the Whenever gem.

Rake tasks can be called from Cron, so we'll use the Whenever Gem to automatically add our job to our Crontab during deployment, and keep our schedule in change control.