har-to-openapi VS openapi-devtools

Genson-js is used to merge JSON Schema objects. Essentially there are 5 schemas that we care about in each request, for request bodies, request headers, response bodies, response headers, and query parameters. Each endpoint (which may or may not be parameterised) has only one schema for each of these values.

The idea for a crawler is a good one. The core logic that handles spec generation is decoupled from everything else, so it can be extracted into its own library.

But there are approaches that exist for this already, such as har-to-openapi.

https://github.com/jonluca/har-to-openapi

Not specific to the CPG realm, but I've had some very good luck working with the OpenAPI-devtools Chrome Extension[1] (previous discussion here on hackernews[2]) to discover the underlying APIs of various sites that I want to scrape data from.

[1] https://github.com/AndrewWalsh/openapi-devtools

[2] https://news.ycombinator.com/item?id=38012032

As a follow up, the algorithm that powers this makes use of the chrome.devtools.network API. Specifically it passes the Request object that is in the HAR 1.2 archive format.

So if you can pass the equivalent of that in Firefox/other browsers to the insert method and switch things up a bit, it should be relatively straightforward. I will think about pulling out the core logic into its own lib.

https://developer.chrome.com/docs/extensions/reference/devto...

https://developer.chrome.com/docs/extensions/reference/devto...

https://github.com/AndrewWalsh/openapi-devtools/blob/main/sr...