cloudflare-scrape VS ebayScraper

CloudFlare is really good at blocking bots. Try a residential proxy (or host it at home) and change the User Agent Header. But it’s really hard. There are some projects e.g. https://github.com/Anorov/cloudflare-scrape but as fare as I know there are all outdated.

I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.

I've attempted to use cfscrape with the URL and headers of a private_api GET request like accounts_for_currency; however, I get 403 forbidden errors despite that. Example below:

If you ever want to expand on this project more, you might enjoy looking at my implementation of an eBay Scraper I made last year: https://github.com/driscoll42/ebayMarketAnalyzer You can see the code I used to specify a specific search to scrape eBay for those instead of needing to put the specific search URL, also filters based on price. The main issue you'll run into sooner or later are CAPTCHAs eBay added earlier this year.

Except they're not, I speak from personal experience. I built a scraper for eBay to analyze sales data and a few months ago eBay added CATPCHAs to the site which prevented my tool from working. They were more complex than the one OP is working on, but still CAPTCHAs. Further I got several emails from other eBay scrapers asking me if I was working on a solution around it. CAPTCHAs aren't perfect but they do work to prevent a large segment of people from scraping a site. If eBay had had CAPTCHAs from the beginning my project never would have started at all.

Here's the repo! https://github.com/driscoll42/ebayScraper I'd love any suggestions to improve. It makes sense on the background/thicker lines, though the image size is a Tom's Hardware thing, by default they're much larger. Example

Source Code for Data Scraping: https://github.com/driscoll42/ebayScraper

No but I'll check that out. I just wrote a pythong script, primarily calling a url with requests and then using beautifulsoup to parse the data. Here's a link if you want to look at it: https://github.com/driscoll42/ebayScraper

Source Code: https://github.com/driscoll42/ebayScraper