pod-registry VS skyscraper

babashka supports html parsing through pods:

https://github.com/babashka/pod-registry

Pods can be written in any language and they can expose functions to babashka by implementing a protocol.

One pod exposing HTML parsing is:

https://github.com/retrogradeorbit/bootleg

Here is an example of how to use that:

https://github.com/babashka/pod-registry/blob/master/example...

This is not how babashka (or the underlying sci interpreter) works: babashka doesn't transpile to another language, but directly executes the code (via an intermediate analyzed representation). Babashka (and sci) are implemented on the JVM in Clojure itself and can leverage libraries directly from that ecosystem, without re-implementing everything for a different host than the JVM. A bb (or sci) program consists of pre-compiled Clojure code (via GraalVM native-image) and interpreted glue code. It aims to be as compatible as possible with JVM Clojure. Babashka integrates with tools.deps.alpha and can execute several existing Clojure libraries "as is" (https://github.com/borkdude/babashka/blob/master/doc/projects.md). It aims to be as compatible as possible with JVM Clojure. With the introduction of pods (https://github.com/babashka/pod-registry) it is able to leverage libraries from other ecosystems as well.

Yes!

My Clojure scraping framework [0] facilitates that kind of workflow, and I’ve been using it to scrape/restructure massive sites (millions of pages). I guess I’m going to write a blog post about scraping with it at scale. Although it doesn’t really scale much above that – it’s meant for single-machine loads at the moment – it could be enhanced to support that kind of workflow rather easily.

[0]: https://github.com/nathell/skyscraper

I plan to port my scraping framework (Skyscraper, https://github.com/nathell/skyscraper) to babashka one day. I’m not sure how easy it will be, though, since it uses core.async (which I believe bb has limited support for) and SQLite via clojure.java.jdbc.

I’ve done a fair share of scraping, and I learned that on a large scale, there are a lot of cross-cutting repetitive concerns. Things like caching, fetching HTML (preferably in parallel), throttling, retries, navigation, emitting the output as a dataset…

My library, Skyscraper [0], attempts to help with these. It’s written in Clojure (based on Enlive or Reaver, both counterparts to Beautiful Soup), but the principles should be readily transferable everywhere.

[0]: https://github.com/nathell/skyscraper