InfluxDB is the Time Series Data Platform where developers build real-time applications for analytics, IoT and cloud-native services in less time with less code. Learn more →
Scrapy Alternatives
Similar projects and alternatives to Scrapy
-
-
-
Zigi
Close all those tabs. Zigi will handle your updates.. Zigi monitors Jira and GitHub updates, pings you when PRs need approval and lets you take fast actions - all directly from Slack! Plus it reduces cycle time by up to 75%.
-
-
-
Pandas
Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
-
playwright-python
Python version of the Playwright testing and automation library.
-
-
Sonar
Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.
-
undetected-chromedriver
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
-
-
-
pyppeteer
Headless chrome/chromium automation library (unofficial port of puppeteer)
-
-
-
Pytorch
Tensors and Dynamic neural networks in Python with strong GPU acceleration
-
-
-
Playwright
Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
-
-
Robot Framework
Generic automation framework for acceptance testing and RPA
-
-
Scout APM
Truly a developer’s best friend. Scout APM is great for developers who want to find and fix performance issues in their applications. With Scout, we'll take care of the bugs so you can focus on building great things 🚀.
NOTE:
The number of mentions on this list indicates mentions on common posts plus user suggested alternatives.
Hence, a higher number means a better Scrapy alternative or higher similarity.
Scrapy reviews and mentions
Posts with mentions or reviews of Scrapy.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2022-11-24.
- Advanced Web Scraping using Python-Scrapy and Splash
-
Ask HN: Best way to keep the raw HTML of scraped pages?
If you weren't already aware, Scrapy has strong support for this via their HTTPCache middleware; you can choose whether to have it actually behave like a cache, choosing to returned already scraped content if matched or merely to act as a pass-through cache: https://docs.scrapy.org/en/2.7/topics/downloader-middleware....
Their OOtB storage does what the sibling comment says about sha1-ing the request and then sharding the output filename by the first two characters: https://github.com/scrapy/scrapy/blob/2.7.1/scrapy/extension...
-
Tool to Scrape Manuals and Sensitive PDFs to Generate Stronger Wordlists for Lateral Movement and Initial Access
Surprised at the name of this project given there is an incredibly popular project called scrapy related to web scraping. This project would really benefit from a rebrand.
- ‘Automate the boring stuff’ — but what do you all actually automate with python
-
What are some cool things you've automated with python?
I was looking for a used cars. I written a scraper using Scrapy, that was gathering all new offers, filtered by my criteria, every hour. Then it was sending me nicely formatted email.
-
Web Search and Scrape
scrapy a python web scraping library
-
13 ways to scrape any public data from any website
scrapy is a high-level webscraping framework designed to scrape data at scale and can be used to create a whole ETL pipeline.
-
Big News at The Extract Summit in London
Of course, the scrapy framework was in the spotlight. If you are interested, you can find the solution is here.
-
estela, an OSS elastic web scraping cluster
[1] https://scrapy.org
-
What have you automated using Python?
One more useful project I forgot. I was looking for a used cars. I written a scraper using Scrapy, that was gathering all new offers, filtered by my criteria, every hour. Then it was sending me nicely formatted email.
-
A note from our sponsor - InfluxDB
www.influxdata.com | 28 Nov 2022
Stats
Basic Scrapy repo stats
135
45,209
9.0
6 days ago
scrapy/scrapy is an open source project licensed under BSD 3-clause "New" or "Revised" License which is an OSI approved license.
Build time-series-based applications quickly and at scale.
InfluxDB is the Time Series Data Platform where developers build real-time applications for analytics, IoT and cloud-native services in less time with less code.
www.influxdata.com