Kargo automates promotion across dev, staging, and prod with approval gates and verification. Open source, built by the team behind Argo CD. Download now. Learn more →
Top 23 web-scraping Open-Source Projects
-
Project mention: Svipall alternatives - firecrawl, crawl4ai, Scrapling, reader, and playwright-mcp | libhunt.com/r/svipall | 2026-09-12
Closest overlap: both turn pages into LLM-ready Markdown and crawl sites. Firecrawl is built around a hosted API with a self-hosted option; Svipall is a single Rust binary that only runs locally, with no account or API key, and speaks MCP, CLI and local REST.
-
AppSignal
AppSignal knows why the f*#k it crashed. Stop vibe-debugging. Every exception, every backtrace, grouped so you see patterns, not noise.
-
Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
Project mention: Svipall alternatives - firecrawl, crawl4ai, Scrapling, reader, and playwright-mcp | libhunt.com/r/svipall | 2026-09-12Scrapling does adaptive parsing, fetchers and spiders in Python, with selectors that survive some redesigns. Svipall fingerprints selectors per domain for the same reason, but ships as a binary aimed at an agent (MCP tools) rather than at code you write.
-
Project mention: Why everyone is talking about loop-engineering and how is it changing agentic ai workflows? Claude Code and Web Scraping examples | dev.to | 2026-06-10
Think about what a mature scraping project already contains. There is a schema that every item must validate against. There are field coverage thresholds, because a run where only 60% of products have prices is a failed run no matter what the exit code says. There are expected item counts, error rate ceilings, and finish reason checks. In the Scrapy world we even have a dedicated framework for all of this, and I wrote about it earlier this year in my post on giving spidey-senses to your spiders with Spidermon. Here is the reframe that I cannot stop thinking about: a Spidermon monitor suite is a rubric. Our community spent a decade encoding "what good data looks like" into machine-checkable criteria, because silent failure is scraping's oldest enemy, the spider that runs green for three weeks while quietly shipping garbage. We built the evaluator long before we had a generator capable of acting on its feedback. Every other field adopting loop engineering has to invent its definition of done from scratch. We just have to plug ours in. The missing piece was never detection. It was what happens after detection, which until now was a human reading an alert, opening the site, sighing at the redesign, and rewriting selectors. Models like Fable 5, which Anthropic says can work autonomously far longer than any previous Claude model, are finally good enough to sit inside that gap. John Rooney saw early versions of this pattern when he built scraping agents for 30 days, and the lesson that stuck with me from his series is that agents fail not from lack of capability but from lack of structure around them. Loops are that structure.
-
Project mention: AI-website-cloner-template: Clone any website using AI coding agents | news.ycombinator.com | 2026-06-24
-
changedetection.io
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!
Project mention: Svipall alternatives - firecrawl, crawl4ai, Scrapling, reader, and playwright-mcp | libhunt.com/r/svipall | 2026-09-12Partial overlap: Svipall's web_watch monitors a whole page or one CSS region while the server runs. changedetection.io is dedicated to that job with a UI and notification integrations; page monitoring is one tool of many in Svipall.
-
-
crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Project mention: Svipall alternatives - firecrawl, crawl4ai, Scrapling, reader, and playwright-mcp | libhunt.com/r/svipall | 2026-09-12Crawlee is a Node library for building scrapers with browser automation and proxy handling. Svipall is not a library: it is a server and CLI an agent calls, with bounded crawls and output written straight to CSV, JSON or Markdown files.
-
Kargo
Stop Scripting Promotions. Start Shipping with Kargo. Kargo automates promotion across dev, staging, and prod with approval gates and verification. Open source, built by the team behind Argo CD. Download now.
-
Douyin_TikTok_Download_API
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
-
maxun
🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥
-
Skill_Seekers
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
Project mention: Turn Docs, Code and PDFs into Claude AI Skills in Minutes | news.ycombinator.com | 2025-11-07 -
SeleniumBase
APIs for browser automation, testing, and bypassing bot-detection. Includes CDP Mode: A stealthy configuration for chromium that passes every bot detection test.
-
crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Project mention: Launching Crawlee for Python v1.0 to simplify building web scrapers and crawlers | news.ycombinator.com | 2025-09-30 -
-
-
firecrawl-mcp-server
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.
Project mention: Show HN: MCP-recorder – VCR.py for MCP servers. Record, replay, verify | news.ycombinator.com | 2026-03-06--target https://mcp.devhelm.io
It works with both HTTP and stdio transports. Scenarios are defined in YAML so it works with MCP servers in any language, and there's a pytest plugin if you want tighter integration. Secret redaction and environment variable interpolation are built in.
To make sure this actually works on real codebases, I submitted several PRs to production MCP servers: monday.com's MCP server (https://github.com/mondaycom/mcp/pull/222), Tavily's MCP server (https://github.com/tavily-ai/tavily-mcp/pull/113), and Firecrawl's MCP server (https://github.com/firecrawl/firecrawl-mcp-server/pull/175). They went from zero schema coverage to full tool surface verification with a clean schema diff available on each tool change. One big benefit is that you can do verification and replay with no API keys — deterministic responses, no live requests to real servers.
I wrote up a deeper dive into the schema drift problem and the VCR pattern for MCP here: https://devhelm.io/blog/regression-testing-mcp-servers
-
Is this just devtools protocol wrapped by an MCP? I've been doing this with go-rod for two years...
https://github.com/go-rod/rod
-
pydoll
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
-
trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Project mention: I built a local, keyless Firecrawl for Claude Code — here's why published: false | dev.to | 2026-07-18Main-content only. Extraction runs through trafilatura, so nav/ads/chrome are gone before anything reaches the model.
-
curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
Project mention: Only 18.5% of the web is open. What an audit of 24,898 sites tells scraper engineers | dev.to | 2026-09-03The mechanism is that your client announces its supported TLS versions, cipher suites and extensions in the ClientHello, and different TLS libraries do this differently. Python's requests on OpenSSL, Go's crypto/tls and Chrome's BoringSSL all produce distinguishable handshakes, so a site can tell "Chrome user agent, OpenSSL handshake" is lying before it reads a single header. JA3, from Salesforce in 2017, turned this into an MD5 hash of those fields in order. Chrome then started randomising the order of its ClientHello extensions in January 2023, which changed the JA3 hash on every connection, and JA4 from FoxIO replaced it by sorting the fields before hashing and producing a human-readable string. Clients like curl-impersonate exist specifically to reproduce a browser's handshake from a non-browser client.
-
skills
Browser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms when stuck. Parallel multi-task execution, independent multi-session operation, isolated multi-account browsing. (by browser-act)
Project mention: How BrowserAct Fixed the Stale-Selector Failures Breaking My Browser Tasks | dev.to | 2026-07-31Skill source: browser-act/skills. Installation details: docs/installation.md.
-
google-maps-scraper
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place (by gosom)
Project mention: gmapsscraper-js VS google-maps-scraper - a user suggested alternative | libhunt.com/r/gmapsscraper-js | 2026-07-17 -
wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
-
Project mention: How to Build a LinkedIn Outreach Pipeline (Without Getting Your Account Banned) | dev.to | 2026-06-12
What worked was switching to Patchright, a Playwright fork that patches those patterns at the Chrome DevTools Protocol level, before any page JavaScript runs. It was a one-line replacement: import { chromium } from 'patchright' and the rest of the Playwright API stays the same. We paired that with installing real google-chrome-stable in the image instead of relying on bundled Chromium, which gives a more believable TLS handshake. LinkedIn's edge can fingerprint the handshake itself through JA3 and JA4, and bundled Chromium has a slightly different signature from the Chrome a real person runs.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
web-scraping discussion
web-scraping related posts
-
Svipall alternatives - firecrawl, crawl4ai, Scrapling, reader, and playwright-mcp
9 projects | 12 Sep 2026 -
Three small API-backed actors I've shipped recently (Docker Hub, lead extraction, SEC Form 4)
-
prefill and default look the same in Console. Only one of them lets an agent skip your required field.
-
Your headless browser has as many cookie jars as it has CDP connections
-
Fingerprinting a website's tech stack from a single HTTP request (no headless browser)
-
Show HN: Turn any website into a CLI for AI agents (142x fewer tokens than HTML)
-
I built a local, keyless Firecrawl for Claude Code — here's why published: false
-
A note from our sponsor - Kargo
akuity.io | 13 Sep 2026
Index
What are some of the best open-source web-scraping projects? This list will help you:
| # | Project | Stars |
|---|---|---|
| 1 | firecrawl | 179,332 |
| 2 | Scrapling | 79,942 |
| 3 | Scrapy | 64,304 |
| 4 | ai-website-cloner-template | 34,160 |
| 5 | changedetection.io | 33,812 |
| 6 | Scrapegraph-ai | 30,835 |
| 7 | crawlee | 25,764 |
| 8 | Douyin_TikTok_Download_API | 20,056 |
| 9 | maxun | 17,425 |
| 10 | Skill_Seekers | 14,951 |
| 11 | SeleniumBase | 13,007 |
| 12 | crawlee-python | 9,512 |
| 13 | awesome-web-scraping | 8,148 |
| 14 | autoscraper | 7,961 |
| 15 | firecrawl-mcp-server | 7,444 |
| 16 | rod | 7,096 |
| 17 | pydoll | 7,082 |
| 18 | trafilatura | 6,802 |
| 19 | curl_cffi | 6,485 |
| 20 | skills | 5,848 |
| 21 | google-maps-scraper | 5,788 |
| 22 | wigolo | 5,161 |
| 23 | patchright | 4,544 |
Sponsored
AppSignal knows why the f*#k it crashed.
Stop vibe-debugging. Every exception, every backtrace, grouped so you see patterns, not noise.
www.appsignal.com