-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
What you want to use instead, is a web crawling framework like scrapy, which provides methods and classes to deal with all the common web scraping requirements. It has functions for pagination, it supports callbacks for using different parsers for different sub sites, it provides link extractors to find and follow urls, asynchronous request handling, logging, automatic request throttling, file exports for your results and many more. In short, it was written to provide all the tools you need to make writing web scrapers as comfortable and easy as possible.
Related posts
-
Scrapy: A Fast and Powerful Scraping and Web Crawling Framework
-
Implementing case sensitive headers in Scrapy (not through `_caseMappings`)
-
Dicas para projetos usando web scraping
-
Best tools to use for web scraping ??
-
I'm using python to scrape web page content and extract keywords, how can I make it faster to process?