What would you like to see next in Scrapy?

• Scrapy

  180 50,954 9.6 Python

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  

Thanks for this initiative. I would like to see a built-in backoff mechanism per request. Currently the solution with custom download delays per domain is used, which is described here. But it would be helpfull to have some mechanism when I can just say "I want this requests be sent not early than N seconds(minutes, etc.)", and in the meantime to process other requests.

• scrapy-inline-requests

  3 110 0.0 Python

  A decorator to write coroutine-like spider callbacks.

I think it would be great to have it built-in in Scrapy as well. In any case, have you looked at https://github.com/rmax/scrapy-inline-requests ? It seems to do what you are looking for.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• scrapy-playwright

  11 837 7.8 Python

  🎭 Playwright integration for Scrapy

  

Eugenio (Scrapy core developer) is currently working on https://github.com/elacuesta/scrapy-playwright

Suggest a related project