got-scraping

HTTP client made for scraping based on got. (by apify)

Got-scraping Alternatives

Similar projects and alternatives to got-scraping

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better got-scraping alternative or higher similarity.

got-scraping reviews and mentions

Posts with mentions or reviews of got-scraping. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2021-09-13.
  • How to Crawl the Web with Scrapy
    7 projects | news.ycombinator.com | 13 Sep 2021
    While I agree that Scrapy is a great tool for beginner tutorials and easy entry into scraping, it's becoming difficult to use it in real world scenarios because almost all the large players now employ some anti-bot or anti-scraping protection.

    A great example above all is Cloudflare. You simply can't convince Cloudflare you're a human with Scrapy alone. Scrapy has only experimental support of HTTP2 and does not support proxies over HTTP2 (https://github.com/scrapy/scrapy/issues/5213). Yet, all browsers use HTTP2 now, which means all normal users use HTTP2... You get the point.

    What we use now is Got Scraping (https://github.com/apify/got-scraping). It's a special purpose extension of Got (HTTP client with 18 mil weekly downloads) that masks its HTTP communication as if it was coming from a real browser. Of course, this will not get you as far as Puppeteer or Playwright (headless browsers), but it improved our scraping tremendously. If you need a full crawling library, see the Apify SDK (https://sdk.apify.com) which uses Got Scraping under the hood.

  • Show HN: Web scraping focused HTTP client for Node.js
    2 projects | news.ycombinator.com | 6 Aug 2021

Stats

Basic got-scraping repo stats
3
375
6.9
about 1 month ago
The modern identity platform for B2B SaaS
The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
workos.com