pup VS gron

And lnk=$(curl -s https://www.selfstudys.com$url |grep "PDFFlip" | cut -d '"' -f 6) to lnk=$(curl -s https://www.selfstudys.com$url | pup "div#PDFF attr{source}" ) here pup will print content of source attribute from div tag with id PDFF i dont know that much about html & css so this is what i came up with. but i am sure you can also select class & make list of suburls from them. check out the video from bugswriter on pup or read docs from git hub for more info github link: https://github.com/ericchiang/pup

jq is pretty amazing. If you are comfortable with its jquery-like CSS selector syntax, then I should also mention a couple similar cli utilities that apply it to HTML: htmlp and pup.

Regex is not a great tool for parsing web pages. Open up a browser dev tools window and select a bit of the page. Right click > copy... XPath expression or CSS selector. A proper web scraping tool will accept either of those. No muss, no fuss. You can even use simple command line tools: xpath or pup

This is HTML to JSON, written in Rust, and there's also pup[1] which I found out about just the other day on HN[2] which uses a very similar syntax (CSS selectors) but outputs HTML and is written in Go.

I can see room for both though it would interesting to have a more detailed comparison to go on (e.g. types of HTML, speed etc).

[1] https://github.com/ericchiang/pup

[2] https://news.ycombinator.com/item?id=33805732

It looks like the project became inactive for a bit and there are alternatives such as htmlq, etc. https://github.com/ericchiang/pup/issues/150

Another tool worth mentioning is pup - it can produce JSON output which means you can pipe it to jq

gron (https://github.com/tomnomnom/gron) to transform it and query and then invert the transformation?

It buffers all of its output statements in memory before writing to stdout:

https://github.com/tomnomnom/gron/blob/master/main.go#L204

Have you tried `gron`?

It converts your nested json into a line by line format which plays better with tools like `grep`

From the project's README:

▶ gron "https://api.github.com/repos/tomnomnom/gron/commits?per_page..." | fgrep "commit.author"

json[0].commit.author = {};

json[0].commit.author.date = "2016-07-02T10:51:21Z";

json[0].commit.author.email = "[email protected]";

json[0].commit.author.name = "Tom Hudson";

https://github.com/tomnomnom/gron

It was suggested to me in HN comments on an article I wrote about `jq`, and I have found myself using it a lot in my day to day workflow

> So all I want is a tool to go from json => line oriented and I will do the rest with the vast library of experience I already have at transformations on the command line.*

The tool for that is likely https://github.com/tomnomnom/gron

If JQ is too much, see GRON &| Miller

gron transforms JSON into discrete assignments to make it easier to grep for what you want https://github.com/tomnomnom/gron

Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON https://github.com/johnkerl/miller

And jless [1] and gron [2].

This is the first I'm hearing of gron, but adding here for completeness sake. Meanwhile, JSON seems to be becoming a standard for CLI tools. Ideal scenario would be if every CLI tool has a --json flag or something similar, so that jc is not needed anymore.

[1] https://jless.io/

[2] https://github.com/tomnomnom/gron