-
-
scrapy-proxycrawl-middleware
Discontinued Scrapy middleware interface to scrape using ProxyCrawl proxy service
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
Alternatively if you want to use scrapy there's a brilliant API addition called scrapyRT which wraps http API on your scrapy project.
You can use Scrapy middleware by ProxyCrawl to get started and scale at speed without the hassle of any infrastructure cost. Here is a link to it on GitHub. You will need new data often, so automating it with Airflow would be the perfect option.
NOTE:
The number of mentions on this list indicates mentions on common posts plus user suggested alternatives.
Hence, a higher number means a more popular project.
Related posts
-
PHP Doesn't Suck Anymore
-
Developed Python CLI to easily download Dutch point cloud data
-
Unlocking Profit Potential: Building an Arbitrage Betting Client with Hexagonal Architecture in Golang
-
Bountysource Stole at Least $17,000 from Open Source Developers
-
Fnug runs all your lints, tests and commands at once, in the terminal