A native Rust port of Google's robots.txt parser and matcher C++ library.
Why do you think that https://github.com/let4be/crusty-core is a good alternative to robotstxt
A native Rust port of Google's robots.txt parser and matcher C++ library.
Why do you think that https://github.com/let4be/crusty-core is a good alternative to robotstxt