Do you think we need an open-source web scraping monitoring tool?

• scrapy-test

  1 2 10.0 Python

  framework to test scrapy spiders (by Granitosaurus)

I wrote scrapy-test as a proof of concept for validating live pages for scrapy spiders if you're looking for some reference but if you're not using scrapy I'd recommend just adding validation tests using data validation tools like cerberus which is super underrated. I cover popular data validation techniques on this short blog I wrote if you want to learn more.

• Cerberus

  3 3,106 7.4 Python

  Lightweight, extensible data validation library for Python (by pyeve)

  

I wrote scrapy-test as a proof of concept for validating live pages for scrapy spiders if you're looking for some reference but if you're not using scrapy I'd recommend just adding validation tests using data validation tools like cerberus which is super underrated. I cover popular data validation techniques on this short blog I wrote if you want to learn more.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project