powerpage-web-crawler VS curl-impersonate

it depends. for no-code solution, please check [powerpage-web-crawler](https://github.com/casualwriter/powerpage-web-crawler) for crawling blog/posts.

Just code a portable lightweight web crawler using Powerpage. Powerpage Web Crawler is a portable javascript-application running with Powerpage. It is coded by vanilla javascript in about 350 lines codes, without any dependency.

may check powerpage-web-crawler, whick a simple yet powerful crawler for blogs or web page.

Powerpage Web Crawler (350 line of code)

> us[e] Akamai to block scraping

Would https://github.com/lwthiker/curl-impersonate help? Haven’t tried with Akamai, but did help with another widely used CDN that shall remain unnamed (but has successfully infused me with burning hate for their products after a couple of years’ worth of using an always-on VPN to bypass Internet censorship and/or a slightly unusual browser).

But before that, I had to create a development environment where I could do the coding. I used Docker and created a docker-compose.yml file on my local system to build a container. At first, I did that on an Arm based computer and the first problem appeared. Although RSS-Bridge was working fine, I couldn't get any data, and the reason was that Ko-Fi.com uses Cloudflare CDN. This is something that a lot of people had issues with in the past. RSS-Bridge solves that problem by using a special build of curl that can impersonate the four major browsers: Chrome, Edge, Safari & Firefox. But unfortunately, that library doesn't work well on Arm-based systems, so I had to move to my trusty Intel-based Linux computer.

Curl-Impersonate: https://github.com/lwthiker/curl-impersonate A special build of curl that can impersonate Chrome & Firefox

I haven’t seen a custom build of Wget, but for Curl there is curl-impersonate[1].

[1] https://github.com/lwthiker/curl-impersonate

If you're just trying to pull a file, the curl-impersonate could be a low-effort option.