Woob: interact with the web outside of browsers

• woob

  1 - -

• software-ideas

  3 85 10.0

I think a version of this is what the internet needs but using headless browsers from the client and with a somewhat-centrally curated set of scraper "recipes" if you will. Basically a community curated/updated set of scraper logic per site (yes some trust is required) that essentially provides JSON data and/or APIs based on the site. Even just a neutered HTML equivalent of sites (e.g. amp w/out the Google and ads stuff) would be good.

Since it is all client side, it can be dubbed a "browser" not a "scraper" and one might hope popularity is high enough that active blocking of it is blatantly user hostile. Granted one hopes that, like EasyList and uBO and others have shown, the community can outpace site owners. Not appearing headless (tunneling captchas, literal mousemove events in pseudo-random human-like ways, etc) should be doable.

It's something I have thought about and once dubbed "recapitate" (https://github.com/cretz/software-ideas/issues/82) and plan to revisit. I have seen many versions of this attempted. We need to encourage shared data extraction tools.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• coq

  87 4,594 10.0 OCaml

  Coq is a formal proof management system. It provides a formal language to write mathematical definitions, executable algorithms and theorems together with an environment for semi-interactive development of machine-checked proofs.

  

Perhaps cf. the theorem-proving assistant Coq, in which case the authors almost certainly knew exactly what they were doing, the idea apparently being that it is good and proper for Anglocentrism to make us uncomfortable in light of superior French scholarship. Still, the community has been quietly mulling a name change for a while [1].

[1] https://github.com/coq/coq/wiki/Alternative-names

Suggest a related project