squint VS puppeteer-cluster

Have you considered https://github.com/squint-cljs/squint ?

Personally I couldn't let go of Clojure's other advantages but at least using the syntax would let you step off the syntax churn bus

Interesting, in the CLJS space we recently got two new libraries in this space:

https://github.com/squint-cljs/squint Which is a thin layer on top of JS

https://github.com/squint-cljs/cherry Which is closer to CLJS semantics and data structures but compiles to .mjs files without any advanced optimizations etc.

I've also experimented a bit with Solid. Here's a SolidJS demo in squint: https://github.com/squint-cljs/squint/blob/main/examples/solidjs/src/App.cljs It runs over here: https://squint-cljs.github.io/demos/squint/solidjs/ (The JS output is smaller than the favicon in that project!)

Maybe Squint https://github.com/squint-cljs/squint or Cherry might fit the bill. https://github.com/squint-cljs/cherry

Should you need a minimal sprinkle of JS, you can still generate this server side using squint (which works as a JVM library) as well, while shipping a pretty small JS library to your client.

Sounds like there's some overlap with squint there!

If urlsToScrape happens to be large enough, the memory and processor load from spawning dozens or hundreds of pages can bring a system down quickly. Consider puppeteer-cluster.

You could set up a service with something like playwright, puppeteer puppeteer-cluster, browserless, to access the service internally and serve screenshots of it to the outside user. You'd have to set up probably some kind of web service with the appropriate routes.

To make Unlighthouse fast, we combined this lighthouse binary with the package puppeteer-cluster, which allows for multi-threaded lighthouse scans.

https://github.com/thomasdondorf/puppeteer-cluster is doing queue-like mechanism but it is not using redis.

I've used this library: Puppeteer cluster