xidel VS xpe

You could try Xidel[1]. It supports JSON, XML and HTML using XPath/XQuery 3.1

It has some extensions to the standard that are pretty nice (JSONiq, CSS selectors, html “template” matching), but you can limit it to just standard XPath/XQuery if you like.

I recommend getting the nightly v .99 build if you give it a try, the stable .98 version is pretty old and I’ve had no issues with .99

1. https://www.videlibri.de/xidel.html

Windows binary of Xidel (https://github.com/benibela/xidel) a commandline tool to download and extract data from HTML pages

Xidel for extraction and pagination

Download an entire subreddit to JSON Lines with Xidel:

I believe that you can't read from a url with batch alone. you'll need a helper app like xidel https://www.videlibri.de/xidel.html

I have not played with this, but it looks like xidel might allow you to do this in Bash. jc also has a URL string parser that could be used in such a script.

I built Xidel for such authenticated requests:

xpe - a commandline xpath parser. xpaths are better than css queries for getting at specific html elements in the DOM. Compared to other parsers, this one is easier to use, and supports html.

I use the heck out of xpe. It's a super simple command-line xpath parser using lxml in python.

If you like xmllint, you might like xpe. It's more user-friendly.

xpe - a commandline xpath parser. I made this after trying to use xpaths for web automation in bash, and not finding anything that worked.

For commandline xpath parsing for simple commandline web automation, xpe is pretty handy. It's a real simple python script, but it scratches that itch, and it's only a pip install away.