Dosage Alternatives

dosage reviews and mentions

• Search web freely from command line?
  2 projects | /r/commandline | 1 Apr 2021

  What fundamentally is an HTML parser? I read Regex is not sufficient to parse HTML. What basic design feature does an HTML parser have that allows this? I mean, what procedures does it use to recognize content between tags, for example? For one thing, tags in HTML can nest. That's enough to make regular expressions insufficient to deal with it. Now, most regex engines can do more than what regular expressions are supposed to be, but the regex syntax itself is both unfriendly to this sort of thing, and complicated regexes can really slow down the engine. For another thing, the HTML you can find in the wild is exceptionally far from being well-behaved. Leaving aside the mess that has accumulated over the decades, just HTML5, for example, allows for tags to stand on their own. That is, a closing is not required, as it's implicitly added by the parser when encountering the next block-level element. If you are dealing with anything of the sort, a regex-based solution will get exceptionally messy, exceptionally quick. What you can use them for is chunking the input - as a lexer - or extracting snippets of text out of the page. Even then, I'd rather write b.select(".search-results > div > a:nth-of-type(1)") than something like s/]*class="[^"]*search-results[^"]*"[^>]*>/; s/]*>]*href="([^"]+)"[^>]*>([^<]+).*/ (not an exaggeration): the latter is not only much more complex, but also frail. You say bash is by its nature inadequate for this. How come? Does it not have good string handling and processing capabilities, for example? Nope. Bash is first and foremost a shell. It being a general-purpose programming language comes after, and it shows: it doesn't just make simple things complicated, it's also full of pitfalls (e.g., " vs ', how they behave with variable and tilde expansion, splitting of parameters, $@ vs $*...). Even just "bash scripting" is ambiguous as hell, because there are subtle and less-than-subtle differences between bash, sh and zsh, between different versions and implementations of those binaries. I'm no slouch when it comes to shell-wrangling - I like a challenge - but I still shellcheck my work, which doesn't stop me from running into issues when using scripts I wrote on a different computer. If I decided to use a real HTML parser, say one in Python, I want to find a way to pass the output directly, after executing curl, without having to save a file. Pass the output to what? You can just do whatever - print the search results? - from inside Python itself, and you can always write a shell wrapper if you want to do something with the output. I'm currently researching how to access the last output of a bash command, after it's been executed. I don't believe it's saved anywhere. A lot of commands can fill up the scrollback of your terminal, or just never end. Trying to grab the output of the last command would mean trying to grab the output of those commands, which would just pointlessly fill your disc.