Meet Crusty, Fast Scalable && Polite Broad Web Crawler Built With Rust!

This page summarizes the projects mentioned and recommended in the original post on reddit.com/r/rust

Our great sponsors
  • Scout APM - Less time debugging, more time building
  • SonarQube - Static code analysis for 29 languages.
  • SaaSHub - Software Alternatives and Reviews
  • crusty

    Broad Web Crawler

    You can check Crusty at https://github.com/let4be/crusty

  • crusty-core

    A small library for building fast and highly customizable web crawlers

    While I was working on Crusty I realized that the web crawling core is universal and spent some time separating crawling logic into a library https://github.com/let4be/crusty-core

  • Scout APM

    Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.

  • robotstxt

    A native Rust port of Google's robots.txt parser and matcher C++ library.

    There is https://github.com/Folyd/robotstxt which passes all tests of the google open source robots.txt parser.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts