phantomgo
CycleTLS
| phantomgo | CycleTLS | |
|---|---|---|
| 1 | 3 | |
| 62 | 802 | |
| - | - | |
| 10.0 | 6.4 | |
| over 7 years ago | 29 days ago | |
| Go | Go | |
| - | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
phantomgo
-
Is it possible to scrape a website protected by Cloudflare?
And can't use https://github.com/k4s/phantomgo since it requires a local binary and the backend is living in a serverless provider.
CycleTLS
-
Is it possible to scrape a website protected by Cloudflare?
A lot of websites nowadays add fingerprint checking. So even if you fake the headers - it wouldn't help, as cloudflare still knows you are making request from go / python / whatever. So there is a lib to spoof fingerptint - https://github.com/Danny-Dasilva/CycleTLS . It may work for you .
- Curl’s TLS Fingerprint
-
Stack under attack: what we learned about handling DDoS attacks
While it can still be spoofed using several packages , e.g. https://github.com/Danny-Dasilva/CycleTLS it can still provide a meaningful/easy-to-manipulate signal.
Moreover, most bots conducting L7 DDoS don't use real/headless browsers in order to be able to scale their attack, so it's highly likely they'll have a discriminating/inconsistent TLS fingerprint.
This can also be done directly in Fastly using e.g. https://developer.fastly.com/reference/vcl/variables/client-...
Another approach to proactively flag malicious IPs is to scrape free proxies. Indeed, most DDoS leverage lot of cheap/know bad IPs. It's frequent to see these free proxies in these attacks.
What are some alternatives?
go-cloudflare-scraper - A golang http.Transport layer that uses Otto to solve Cloudflare challenges
gost - GO Simple Tunnel - a simple tunnel written in golang
surf - Stateful programmatic web browsing in Go.
chromedp - A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.
mimic - Mimic chromium's HTTP/HTTP2 and TLS implementations.
colly - Elegant Scraper and Crawler Framework for Golang
Ponzu - Headless CMS with automatic JSON API. Featuring auto-HTTPS from Let's Encrypt, HTTP/2 Server Push, and flexible server framework written in Go.