Top 10 Webcrawler Open-Source Projects

• crawlab

  4 10,872 6.0 Go

  Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

  

• apollo

  7 1,360 0.0 Go

  A Unix-style personal search engine and web crawler for your digital footprint. (by amirgamil)

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• scrapyrt

  3 817 6.8 Python

  HTTP API for Scrapy spiders

  

• SpiderSuite

  8 543 6.1

  Advance web security spider/crawler

Project mention: Show HN: SpiderSuite version 1.0.4. Whats new? | news.ycombinator.com | 2023-09-17

• Rcrawler

  2 344 0.0 R

  An R web crawler and scraper

• krawler

  0 130 0.0 Kotlin

  A web crawling framework written in Kotlin

• wbot

  2 17 4.6 Go

  A simple & efficient web crawler.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• Abosar

  1 12 8.1 Python

  অবসর 📚 A collection of short Bengali stories web scraped from various Bengali eMagazines and eNewspapers.

• Spidey

  2 11 9.5 C#

  A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.

• Edupedu_crawler

  1 2 10.0 Python

  Un mic crawler in python care preia ultimele informatii despre profesori si greva de pe Edupedu si le comprima cu ajutorul lui BingAI direct in README

Project mention: Am facut un "pseudo-live-feed" al grevei | /r/Romania | 2023-05-28

Link repo: https://github.com/Mike4544/Edupedu_crawler

Webcrawler related posts

• Self-hosted web scraper?

  4 projects | /r/selfhosted | 3 Jan 2023

• I need data from a website. It is viable to create an API that scrapes the website and returns the data on an endpoint?

  2 projects | /r/dotnet | 20 Dec 2022

• Can R do recursive web crawling?

  1 project | /r/RStudio | 22 Dec 2022

• CI/CD in Action: Manage auto builds of large open-source projects with GitHub Actions?

  4 projects | dev.to | 20 Oct 2022

• CI/CD in Action: How to use Microsoft's GitHub Actions in a right way?

  2 projects | dev.to | 13 Oct 2022

• A small web crawler - WBot

  2 projects | /r/golang | 14 May 2022

• GitHub - amirgamil/apollo: A Unix-style personal search engine and web crawler for your digital footprint.

  1 project | /r/devopsish | 28 Jul 2021
• A note from our sponsor - InfluxDB
  www.influxdata.com | 20 May 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com