xidel VS fq

You could try Xidel[1]. It supports JSON, XML and HTML using XPath/XQuery 3.1

It has some extensions to the standard that are pretty nice (JSONiq, CSS selectors, html “template” matching), but you can limit it to just standard XPath/XQuery if you like.

I recommend getting the nightly v .99 build if you give it a try, the stable .98 version is pretty old and I’ve had no issues with .99

1. https://www.videlibri.de/xidel.html

Windows binary of Xidel (https://github.com/benibela/xidel) a commandline tool to download and extract data from HTML pages

Xidel for extraction and pagination

Download an entire subreddit to JSON Lines with Xidel:

I believe that you can't read from a url with batch alone. you'll need a helper app like xidel https://www.videlibri.de/xidel.html

I have not played with this, but it looks like xidel might allow you to do this in Bash. jc also has a URL string parser that could be used in such a script.

I built Xidel for such authenticated requests:

Hey! fq author here. I have a bunch of related tools in the readme https://github.com/wader/fq?tab=readme-ov-file#tools two suggestions: gnu poke and wireshark (can decode lots of more things then just network protocol)

Did someone say let's represent structured data as json? a bit of shameless plug: https://github.com/wader/fq :) It's using a fork of gojq btw!

https://github.com/wader/fq has a REPL and can read JSON. Tip is to use "paste | from_json | repl" in a REPl to paste JSON into a sub-REPL, you can also use `` with fq which is a raw string literal

I do lots of exploratory work in various structure data, in my case often debugging media filea via https://github.com/wader/fq, which mean doing lots of use-once-queries on the command line or REPL. In those cases jq line-friendly and composable syntax and generators really shine.