Top 6 C# Crawler Projects

abot

1 2,072 0.0 C#

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Project mention: Can you build a web crawler in c#? | reddit.com/r/learnprogramming | 2022-04-11

This can be done perfectly well in c#, https://github.com/sjdirect/abot for example.
TumblThree

2 477 8.1 C#

A Tumblr, Twitter and newTumbl Blog Backup Application

Project mention: Twitter archive not working? | reddit.com/r/Twitter | 2022-11-18

There's a github project called TumblThree that can pull tweets and media that have been publicly posted to any account you give it. It can't do private accounts or DMs and it doesn't pull quote tweets, but it gives you a backup of everything publicly available.
SonarLint

www.sonarlint.org
sponsored

Clean code begins in your IDE with SonarLint. Up your coding game and discover issues early. SonarLint is a free plugin that helps you find & fix bugs and security issues from the moment you start writing code. Install from your favorite IDE marketplace today.
Infinity Crawler

0 202 3.2 C#

A simple but powerful web crawler library for .NET
custom-crawler

1 54 10.0 C#

🌌 High productivity semi-automatic crawler generator 🛠️🧰

Project mention: I made a powerful crawler creation tool on c#! | reddit.com/r/csharp | 2022-11-10
Spidey

1 11 0.0 C#

A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.

Project mention: I need data from a website. It is viable to create an API that scrapes the website and returns the data on an endpoint? | reddit.com/r/dotnet | 2022-12-20

Didn't get a chance to reply earlier but depending on what you're trying to do, you might want a web crawler. I have a crawler on Github that I built for scraping in instances where someone doesn't have an API. If you go this route, I suggest doing it as a background task and go off cached data.
Scraper

1 1 10.0 C#

A simplistic web scraper which aims to make scrapping webpages and crawling links faster by utilizing parallel threading. (by 0x78f1935)

Project mention: CLI Scraper | reddit.com/r/csharp | 2022-11-27

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2022-12-20.