SaaSHub helps you find the best software and product alternatives Learn more →
Top 23 Python Webscraping Projects
-
Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
Claude Code can read my files and run my shell, but out of the box it can't actually go get a page from the live web in a way that survives modern anti-bot defenses. A curl from the Bash tool gets you a 403 from anything behind Cloudflare. So I took an existing open-source scraper — D4Vinci/Scrapling — installed it locally, and registered its built-in MCP server with Claude Code. Now the agent has ten tools for pulling the real web: plain fetches, headless-browser fetches, stealth fetches that solve Cloudflare, and screenshots. The scraper isn't mine — I want to be clear about that. What I built is the local install and the MCP integration that hands those capabilities to the agent.
-
AppSignal
Monitoring that respects your time & budget. APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.
-
-
I spent a weekend studying GPT-Researcher, an open-source project with 24,000+ GitHub stars. It builds an autonomous research agent that generates comprehensive reports with citations. The architecture is elegant: multiple specialized agents coordinate through LangGraph, parallel execution speeds up research, and quality gates ensure reliable output.
-
-
-
CloakBrowser
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 30/30 tests passed.
Project mention: A Green Run Is Not a Green Light: Engineering Responsibility in Browser Automation | dev.to | 2026-09-05CloakBrowser belongs primarily in the runtime layer. It provides a source-modified Chromium environment for Playwright, Puppeteer, Selenium, and CDP workflows where conventional headless setups or JavaScript-level patches can become fragile. A more consistent browser gives teams a cleaner foundation for QA, monitoring, browser agents, brand verification, and public-web research.
-
CrossLinked
LinkedIn enumeration tool to extract valid employee names from an organization through search engine scraping
-
Kargo
Stop Scripting Promotions. Start Shipping with Kargo. Kargo automates promotion across dev, staging, and prod with approval gates and verification. Open source, built by the team behind Argo CD. Download now.
-
-
-
-
-
happy to offer a counter of some great products for anti-bot defeat:
https://brightdata.com/
https://www.zenrows.com/
https://www.capsolver.com/
https://scrapfly.io/
hundreds of millions of residential ips, human browser fingerprints, custom browser binaries, auto solve of turnstyle, recaptcha v3, kasada, datadome, AWS WAF, etc if they come up.
-
-
EasyApplyJobsBot
A python bot to automatically apply all Linkedin,Glassdoor, etc Easy Apply jobs based on your preferences. Auto login, auto fill additional questions, apply automatically!
-
-
AutoScraper
Official implement of paper "AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation" [EMNLP 24'] (by EZ-hwh)
-
fortress
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change. (by tiliondev)
Project mention: One Wikipedia page costs your AI agent 68,000 tokens | news.ycombinator.com | 2026-07-10repo and the reproducible benchmark: https://github.com/tiliondev/fortress/tree/main/mcp
-
-
ChatGPT-OpenAI-Smart-Speaker
This AI Smart Speaker uses speech recognition, TTS (text-to-speech), and STT (speech-to-text) to enable voice and vision-driven conversations, with additional web search capabilities via OpenAI and Langchain agents.
-
llm_osint
LLM OSINT is a proof-of-concept method of using LLMs to gather information from the internet and then perform a task with this information.
-
ebayScraper
Scrape all eBay sold listings to determine average/median pricing, plot listings over time with trend lines, and extract to excel
-
yars
Yet Another Reddit Scrapper (without API keys) | Scrap search results, posts and images from subreddits filtered by hot, new etc and bulk download any user's data.
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python Webscraping discussion
Python Webscraping related posts
-
AI scrapers request commented scripts
-
A Comprehensive Guide to TikTok API
-
Ask HN: Who is hiring? (November 2024)
-
CyberScraper 2077 – A OpenAI / Gemini Based Web Scraper
-
CyberScraper-2077 – A LLM Based Web Scraper
-
How To Scrape TikTok in 2024
-
Direction Of The Stock Market
-
A note from our sponsor - SaaSHub
www.saashub.com | 6 Sep 2026
Index
What are some of the best open-source Webscraping projects in Python? This list will help you:
| # | Project | Stars |
|---|---|---|
| 1 | Scrapling | 78,237 |
| 2 | Scrapegraph-ai | 30,546 |
| 3 | gpt-researcher | 29,280 |
| 4 | autoscraper | 7,936 |
| 5 | CyberScraper-2077 | 3,252 |
| 6 | CloakBrowser | 2,249 |
| 7 | CrossLinked | 1,580 |
| 8 | requests-cache | 1,501 |
| 9 | scrapeghost | 1,442 |
| 10 | parsera | 1,354 |
| 11 | mov-cli | 1,298 |
| 12 | scrapfly-scrapers | 1,071 |
| 13 | zimit | 840 |
| 14 | EasyApplyJobsBot | 816 |
| 15 | gazpacho | 769 |
| 16 | AutoScraper | 488 |
| 17 | fortress | 480 |
| 18 | TikTokBot | 417 |
| 19 | ChatGPT-OpenAI-Smart-Speaker | 318 |
| 20 | llm_osint | 316 |
| 21 | ebayScraper | 267 |
| 22 | yars | 220 |
| 23 | par_scrape | 218 |
Sponsored
Monitoring that respects your time & budget
APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.
www.appsignal.com