xidel VS yq

You could try Xidel[1]. It supports JSON, XML and HTML using XPath/XQuery 3.1

It has some extensions to the standard that are pretty nice (JSONiq, CSS selectors, html “template” matching), but you can limit it to just standard XPath/XQuery if you like.

I recommend getting the nightly v .99 build if you give it a try, the stable .98 version is pretty old and I’ve had no issues with .99

1. https://www.videlibri.de/xidel.html

Windows binary of Xidel (https://github.com/benibela/xidel) a commandline tool to download and extract data from HTML pages

Xidel for extraction and pagination

Download an entire subreddit to JSON Lines with Xidel:

I believe that you can't read from a url with batch alone. you'll need a helper app like xidel https://www.videlibri.de/xidel.html

I have not played with this, but it looks like xidel might allow you to do this in Bash. jc also has a URL string parser that could be used in such a script.

I built Xidel for such authenticated requests:

Install jq and yq, and wrap your commands with | yq -y ..

Confusingly there is another project called yq that does exactly what you're suggesting and it's a preprocessor that converts yaml to json and then used jq. https://github.com/kislyuk/yq

yq

I personally find the yq tool from https://github.com/kislyuk/yq much more useful: it has all the same options and formats as `jq` (as it's really a wrapper around jq). Rather than the `yq` in the OP here where only partial functionality exists.

XML data from the API can be parsed and filtered with xq. There may be multiple ways to get it; first try the yq toolset which includes it.

There is also yq [1], which attempts the same for yaml, toml and xml. (And confusingly also contains a binary named "xq" for querying xml, however with a different syntax)

[1] https://github.com/kislyuk/yq