Spidey
Infinity Crawler
Our great sponsors
| Spidey | Infinity Crawler | |
|---|---|---|
| 2 | - | |
| 11 | 238 | |
| - | 2.5% | |
| 9.5 | 5.0 | |
| 11 days ago | 4 months ago | |
| C# | C# | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Spidey
-
I need data from a website. It is viable to create an API that scrapes the website and returns the data on an endpoint?
Didn't get a chance to reply earlier but depending on what you're trying to do, you might want a web crawler. I have a crawler on Github that I built for scraping in instances where someone doesn't have an API. If you go this route, I suggest doing it as a background task and go off cached data.
-
Recursion needed in small crawler
This may be overkill but I have library out there for building web crawlers. Spidey is the library. I'm not suggesting you use it but you could look at it for ideas. It uses a multithreaded, producer/consumer approach that avoids recursion and stack overflow issues. Use a queue, pull from the queue for each url, push new urls on when you find them. Do need to optimize my code a bit more but if it helps at all. But your issue is most likely the fact that you're finding a link to the page you are currently on. HashSet or List of found URLs would solve the issue.
Infinity Crawler
We haven't tracked posts mentioning Infinity Crawler yet.
Tracking mentions began in Dec 2020.
What are some alternatives?
scrapyrt - HTTP API for Scrapy spiders
Downloader - Fast, cross-platform and reliable multipart downloader with asynchronous progress events for .NET applications.
scrapy-sanoma-kuntavaalit2021 - Fetch Sanoma kuntavaalit 2021 data [Moved to: https://github.com/raspi/scrapy-kuntavaalit2021-sanoma]
Sieve - ⚗️ Clean & extensible Sorting, Filtering, and Pagination for ASP.NET Core
SharpNetSH - A simple netsh library for C#
Fluent Command Line Parser - A simple, strongly typed .NET C# command line parser library using a fluent easy to use interface
Colorful.Console - Style your .NET console output!
CodeHub - CodeHub is an iOS application written using Xamarin
OctoLinker - OctoLinker — Links together, what belongs together
CsConsoleFormat - .NET C# library for advanced formatting of console output [Apache]
Netling - Netling is a load tester client for easy web testing.
Open Live Writer - An open source fork of Windows Live Writer