A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.
Why do you think that https://github.com/TurnerSoftware/InfinityCrawler is a good alternative to Spidey
A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.
Why do you think that https://github.com/TurnerSoftware/InfinityCrawler is a good alternative to Spidey