Our great sponsors
-
pyxargs
Command line Python scripting with an xargs-like interface and AWK-like capabilities for data processing and task automation
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
If JQ is too much, see GRON &| Miller
gron transforms JSON into discrete assignments to make it easier to grep for what you want https://github.com/tomnomnom/gron
Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON https://github.com/johnkerl/miller
> I too can never remember jq syntax when I need to. I usually just end up writing a Python script
Same here! That's why for small things I made pyxargs [1] to use python in the shell. In another thread I also just learned of pyp [2] which I haven't tried yet but looks like it'd be even better for this use case.
[1] https://github.com/elesiuta/pyxargs
[2] https://github.com/hauntsaninja/pyp
> I too can never remember jq syntax when I need to. I usually just end up writing a Python script
Same here! That's why for small things I made pyxargs [1] to use python in the shell. In another thread I also just learned of pyp [2] which I haven't tried yet but looks like it'd be even better for this use case.
[1] https://github.com/elesiuta/pyxargs
[2] https://github.com/hauntsaninja/pyp
"Those tried and true commands we were referring to? None other than the usual awk sed cut grep and of course the Unix pipe | to glue them all together. Really, why use a JSON parsing program that only could only do one function (parse JSON) when I could use a combination of tools that, when piped together, could do far more?"
IMHO, drinking the UNIX Kool-Aid means not only using coreutils and BSD userlands but also using the language in which almost all of those programs are written: C. For me, that means gcc and binutils are amongst the "tried and true commands". Also among them is flex. These are found on all the UNIX varieties I use, usually because they are used in compiling the OS. As such, no special installation is needed.
When I looked at jq in 2013, I noticed it used flex and possibly yacc/bison. No idea if it still does.
Using UNIX text processing utilities to manipulate JSON is easy enough. However if I am repeatedly processing JSON from the same source, e.g., YouTube, then I use flex instead of sed, etc. It's faster.
jq uses flex in the creation of a language interpreter intended^1 to process any JSON. I use flex not to create a language interpeter but to process only JSON from a single source. The blog author uses shell script to process JSON from a single source.^2 I think of the use I make of flex as like a compiled shell script. It's faster.
The blog author states than jq is specific to one type of text processing input: JSON. I write a utility that is specific to one source of JSON.
1. Sometimes it's not used as intended, e.g., https://github.com/makenowjust/bf.jq
2. I also used flex to make simple utility to reformat JSON from any source so it's easer to read and process with line-oriented UNIX utilities. Unlike jq and other JSON reformatters it does not require 100% correct JSON; e.g., it can accept JSON that is mixed in with HTML which I find is quite common in today's web pages.
Using awk/sed to parse json seems to be using the wrong tool for the job.
As an alternative to jq with easier to remember syntax, see https://fx.wtf/
Recent discussion: https://news.ycombinator.com/item?id=37567009