dosage VS xidel

What fundamentally is an HTML parser? I read Regex is not sufficient to parse HTML. What basic design feature does an HTML parser have that allows this? I mean, what procedures does it use to recognize content between tags, for example? For one thing, tags in HTML can nest. That's enough to make regular expressions insufficient to deal with it. Now, most regex engines can do more than what regular expressions are supposed to be, but the regex syntax itself is both unfriendly to this sort of thing, and complicated regexes can really slow down the engine. For another thing, the HTML you can find in the wild is exceptionally far from being well-behaved. Leaving aside the mess that has accumulated over the decades, just HTML5, for example, allows for tags to stand on their own. That is, a closing is not required, as it's implicitly added by the parser when encountering the next block-level element. If you are dealing with anything of the sort, a regex-based solution will get exceptionally messy, exceptionally quick. What you can use them for is chunking the input - as a lexer - or extracting snippets of text out of the page. Even then, I'd rather write b.select(".search-results > div > a:nth-of-type(1)") than something like s/]*class="[^"]*search-results[^"]*"[^>]*>/; s/]*>]*href="([^"]+)"[^>]*>([^<]+).*/ (not an exaggeration): the latter is not only much more complex, but also frail. You say bash is by its nature inadequate for this. How come? Does it not have good string handling and processing capabilities, for example? Nope. Bash is first and foremost a shell. It being a general-purpose programming language comes after, and it shows: it doesn't just make simple things complicated, it's also full of pitfalls (e.g., " vs ', how they behave with variable and tilde expansion, splitting of parameters, $@ vs $*...). Even just "bash scripting" is ambiguous as hell, because there are subtle and less-than-subtle differences between bash, sh and zsh, between different versions and implementations of those binaries. I'm no slouch when it comes to shell-wrangling - I like a challenge - but I still shellcheck my work, which doesn't stop me from running into issues when using scripts I wrote on a different computer. If I decided to use a real HTML parser, say one in Python, I want to find a way to pass the output directly, after executing curl, without having to save a file. Pass the output to what? You can just do whatever - print the search results? - from inside Python itself, and you can always write a shell wrapper if you want to do something with the output. I'm currently researching how to access the last output of a bash command, after it's been executed. I don't believe it's saved anywhere. A lot of commands can fill up the scrollback of your terminal, or just never end. Trying to grab the output of the last command would mean trying to grab the output of those commands, which would just pointlessly fill your disc.

You could try Xidel[1]. It supports JSON, XML and HTML using XPath/XQuery 3.1

It has some extensions to the standard that are pretty nice (JSONiq, CSS selectors, html “template” matching), but you can limit it to just standard XPath/XQuery if you like.

I recommend getting the nightly v .99 build if you give it a try, the stable .98 version is pretty old and I’ve had no issues with .99

1. https://www.videlibri.de/xidel.html

Windows binary of Xidel (https://github.com/benibela/xidel) a commandline tool to download and extract data from HTML pages

Xidel for extraction and pagination

Download an entire subreddit to JSON Lines with Xidel:

I believe that you can't read from a url with batch alone. you'll need a helper app like xidel https://www.videlibri.de/xidel.html

I have not played with this, but it looks like xidel might allow you to do this in Bash. jc also has a URL string parser that could be used in such a script.

I built Xidel for such authenticated requests: