crusty-core
spider
Our great sponsors
| crusty-core | spider | |
|---|---|---|
| 1 | 1 | |
| 13 | 630 | |
| - | 18.6% | |
| 0.0 | 9.5 | |
| over 1 year ago | 2 days ago | |
| Rust | Rust | |
| GNU General Public License v3.0 only | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
crusty-core
-
Meet Crusty, Fast Scalable && Polite Broad Web Crawler Built With Rust!
While I was working on Crusty I realized that the web crawling core is universal and spent some time separating crawling logic into a library https://github.com/let4be/crusty-core
spider
What are some alternatives?
async-tungstenite - Async binding for Tungstenite, the Lightweight stream-based WebSocket implementation
colly - Elegant Scraper and Crawler Framework for Golang
console - a debugger for async rust!
Crawler4j - Open Source Web Crawler for Java
crusty - Broad Web Crawler
scraping-with-rust - 👾 scraping hacker news with rust
rustpad - Efficient and minimal collaborative code editor, self-hosted, no database required
rust-bitcoin-indexer - Powerful & versatile Bitcoin Indexer, in Rust
robotstxt - A native Rust port of Google's robots.txt parser and matcher C++ library.
nipper - A Rust crate for manipulating HTML with CSS selectors
chan-downloader - CLI to download all images/webms in a 4chan thread
crab - Python-based Scraping and parsing toolkit