spider
crusty-core
spider | crusty-core | |
---|---|---|
1 | 1 | |
655 | 13 | |
9.6% | - | |
9.5 | 0.0 | |
about 12 hours ago | over 1 year ago | |
Rust | Rust | |
MIT License | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
spider
crusty-core
-
Meet Crusty, Fast Scalable && Polite Broad Web Crawler Built With Rust!
While I was working on Crusty I realized that the web crawling core is universal and spent some time separating crawling logic into a library https://github.com/let4be/crusty-core
What are some alternatives?
colly - Elegant Scraper and Crawler Framework for Golang
async-tungstenite - Async binding for Tungstenite, the Lightweight stream-based WebSocket implementation
Crawler4j - Open Source Web Crawler for Java
console - a debugger for async rust!
scraping-with-rust - 👾 scraping hacker news with rust
crusty - Broad Web Crawler
rust-bitcoin-indexer - Powerful & versatile Bitcoin Indexer, in Rust
rustpad - Efficient and minimal collaborative code editor, self-hosted, no database required
nipper - A Rust crate for manipulating HTML with CSS selectors
robotstxt - A native Rust port of Google's robots.txt parser and matcher C++ library.
chan-downloader - CLI to download all images/webms in a 4chan thread
crab - Python-based Scraping and parsing toolkit