11 best open-source web crawlers and scrapers in 2024

1. crawlee

   1 47 18,408 9.7 TypeScript

   Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

   

   Language: Node.js, Python | GitHub: 15.4K+ stars | link

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. Scrapy

   2 189 57,527 9.7 Python

   Scrapy, a fast high-level web crawling & scraping framework for Python.

   

   Language: Python | GitHub: 52.9k stars | link

4. MechanicalSoup

   3 5 4,778 5.2 Python

   A Python library for automating interaction with websites.

   

   Language: Python | GitHub: 4.7K+ stars | link

5. node-crawler

   4 4 6,766 5.7 TypeScript

   Web Crawler/Spider for NodeJS + server-side jQuery ;-)

   

   Language: Node.js | GitHub: 6.7K+ stars | link

6. Selenium WebDriver

   5 71 32,716 9.9 Java

   A browser automation framework and ecosystem.

   

   Language: Multi-language | GitHub: 30.6K stars | link

7. heritrix3

   6 8 2,999 9.4 Java

   Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

   

   Language: Java | GitHub: 2.8K+ stars | link

8. Apache Nutch

   7 4 3,043 7.0 Java

   Apache Nutch is an extensible and scalable web crawler

   

   Language: Java | GitHub: 2.9K+ stars | link

9. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

10. webmagic

    8 2 11,583 6.3 Java

    A scalable web crawler framework for Java.

    

    Language: Java | GitHub: 11.4K+ stars | link

11. Nokogiri

    9 24 6,202 9.4 C

    Nokogiri (鋸) makes it easy and painless to work with XML and HTML from Ruby.

    

    Language: Ruby | GitHub: 6.1K+ stars | link

12. Crawler4j

    10 3 4,587 0.0 Java

    Open Source Web Crawler for Java

    

    Language: Java | GitHub: 4.5K+ stars | link

13. web-harvest

    11 1 3 - Java

    Web data extraction (web data mining, web scraping) tool. It leverages well proved XML and text processing techologies in order to easily extract useful data from arbitrary web pages. Fork of http://web-harvest.sourceforge.net

    

    Language: Go | GitHub: 11.1k | link

Suggest a related project