wrp VS chromedp

https://github.com/tenox7/wrp allows you to use modern js as well, as long as you can render images

I use this for retrocomputing, in a way, to be very silly! tenox7's WRP [Web Rendering Proxy] basically runs headless Chrome and renders pages to gif/png/jpg and shoves them back at a client browser.

https://github.com/tenox7/wrp

Just like the PowerBook G3 Pismo I have, the iBook uses commonly available 18650 cells, so it's easy to rebuild the battery. With built-in AirPort antennas, you can add an AirPort card and get WiFi (has to be Wireless B though, and sometimes there are issues with encryption so you usually have to run an open access point to connect). Now this is the crazy part, but you can run Web Rendering Proxy on a Raspberry Pi or other, more powerful machine, then use an old browser like Internet Explorer to browse the web. It mostly works, and it's a pretty fun way to get some strange looks in the coffee shop.

Depending on what you're trying to do, you can use a "vintage" web browser and something like Web Rendering Proxy to make the page work.

There are a billion things that you need to consider when building a decent web crawler, especially interacting with pages in the modern web. For example, a lot of content is dynamically loaded by the browser nowadays, and won't show up if you make a simple HTTP request. Open your browser devtools and look at the network tab after you make a request, and you'll see it makes loads of auxiliary requests. Some content is also only loaded after you interact with it (e.g. hover, click). For that reason I'd recommend using something like chromedp and do browser based crawling, even if it's much slower.

You could use something like the chrome dev tools (https://github.com/chromedp/chromedp) or look into JS runtimes in Go and try to handle the I/O yourself.

Thanks. I've found Chromedp [0] to be a good Selenium replacement when programming in Go. Used in conjunction with HeadlessShell [1] you can deploy a Go app into a container and do the testing all within the same container with low overhead.

[0] https://github.com/chromedp/chromedp

Is it node specific? Could you just use https://github.com/chromedp/chromedp and run it as an actually webpage and get the value from that?

It looks like https://github.com/chromedp/chromedp is the most popular option. There's also https://github.com/playwright-community/playwright-go. Any thoughts or advice about using these tools?

The alternative is to drive, for example, Chrome with the DevTools protocol (using something like https://github.com/chromedp/chromedp or https://github.com/raff/godet (this is my version of a DevTools driver).