What do .NET devs use for web scraping these days?

This page summarizes the projects mentioned and recommended in the original post on /r/dotnet
HacktoberFest HTML and CSS Chrome Chromium Web Crawling

WorkOS
Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • PuppeteerSharp

    Headless Chrome .NET API

    • PuppeteerSharp

  • playwright-dotnet

    .NET version of the Playwright testing and automation library.

    • Playwright

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo

  • HtmlAgilityPack

    Html Agility Pack (HAP) is a free and open-source HTML parser written in C# to read/write DOM and supports plain XPATH or XSLT. It is a .NET code library that allows you to parse "out of the web" HTML files.

    • HTMLAgilityPack

  • AngleSharp

    :angel: The ultimate angle brackets parser library parsing HTML5, MathML, SVG and CSS to construct a DOM based on the official W3C specifications.

    • AngleSharp

  • Scrapy

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    • I know this might not be a good answer, as it's not .NET, but we use https://scrapy.org/ (Python).

  • docker-selenium

    Provides a simple way to run Selenium Grid with Chrome, Firefox, and Edge using Docker, making it easier to perform browser automation

    • Selenium with Docker Selenium Grid https://github.com/SeleniumHQ/docker-selenium. Isolated and scalable.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts