Celery lock a variable between two processes

• Scrapy

  180 50,896 9.6 Python

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  

In general celery tasks should be idempotent if possible, for scraping consider if Scrapy might not be more appropriate, it already implements a lot of the rate limiting/retrying you have to replicate in celery yourself. But regarding locking you are right to consider databases/redis since celery workers might run on entirely different machines even. In the case of a paginated scrape with celery, you could schedule another task from within a task itself. To adhere to rate limits you could spawn a delayed task.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project