Any decent web crawler?

• colly

  39 22,165 6.0 Go

  Elegant Scraper and Crawler Framework for Golang

  

I love the lang but seems like it lacks some libs I need. Good example will be this in Go github.com/gocolly/colly

• Crawler4j

  2 4,469 0.0 Java

  Open Source Web Crawler for Java

and Java github.com/yasserg/crawler4j

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• voyager

  3 705 3.0 Rust

  crawl and scrape web pages in rust (by mattsse)

Have you had a look at voyager yet?

• spider

  1 630 9.5 Rust

  The fastest web crawler written in Rust. Maintained by @a11ywatch. (by spider-rs)

  

Suggest a related project