Is it possible to scrape a website protected by Cloudflare?

• colly

  39 22,120 6.0 Go

  Elegant Scraper and Crawler Framework for Golang

  

• surf

  1 1,417 0.0 Go

  Stateful programmatic web browsing in Go. (by headzoo)

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• go-cloudflare-scraper

  2 126 0.0 Go

  A golang http.Transport layer that uses Otto to solve Cloudflare challenges

  

• phantomgo

  1 62 10.0 Go

  a headless browser phantomjs for golang

And can't use https://github.com/k4s/phantomgo since it requires a local binary and the backend is living in a serverless provider.

• CycleTLS

  3 786 6.7 Go

  Spoof TLS/JA3 fingerprints in GO and Javascript

A lot of websites nowadays add fingerprint checking. So even if you fake the headers - it wouldn't help, as cloudflare still knows you are making request from go / python / whatever. So there is a lib to spoof fingerptint - https://github.com/Danny-Dasilva/CycleTLS . It may work for you .

• chromedp

  27 10,341 5.5 Go

  A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.

  

Suggest a related project