Scrapy, a fast high-level web crawling & scraping framework for Python.
The scrapy framework would be a good fit for what you want to do (especially if you plan on hosting the crawler), however the learning curve is quite steep, especially if you don't have much experience using python.
My machine-learning pet project. Part 4. Feeding the dataset to Retinanet
2 projects | dev.to | 26 Jun 2022
I’m a beginner, can i learn web scraping or should I focus on other simpler things
1 project | reddit.com/r/pythontips | 17 Jun 2022
Working as a DS, what tools do you use to scrape data?
2 projects | reddit.com/r/datascience | 9 Jun 2022
My machine-learning pet project. Part 2. Preparing my dataset
2 projects | dev.to | 2 Jun 2022
Django custom management command running Scrapy: How to include Scrapy's options?
1 project | reddit.com/r/codehunter | 1 May 2022