An Introduction to JQ

• jiq

  5 900 0.0 Go

  Discontinued jid on jq - interactive JSON query tool using jq expressions

Something I just learned about the other day was jid [0] to help query the json keys

[0] https://github.com/fiatjaf/jiq

• gron

  64 13,483 0.0 Go

  Make JSON greppable!

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• json-logs

  4 14 0.0 Go

  A tool to pretty-print JSON logs, like those from zap or logrus.

Big fan of JQ. I like it more than the traditional UNIX suite of text manipulation commands, because I get closer to "querying" rather than just filtering. It has really made me rethink where I want "interacting with a computer" to go in the future -- less typing commands, more querying stuff.

I have a few utilities involving JQ that I wrote.

For structured logs, I have jlog. Pipe JSON structured logs into it, and it pretty-prints the logs. For example, time zones are converted to your local time, if you choose; or you can make the timestamps relative to each other, or now. It includes jq so that you can select relevant log lines, delete spammy fields, join fields together, etc. Basically, every time you run it, you get the logs YOU want to look at. https://github.com/jrockway/json-logs. Not to oversell it, but this is one of the few pieces of software I've written that passes the toothbrush test -- I use it twice a day, every day. All the documentation is in --help; I should really paste that into the Github readme.

I am also a big fan of using JQ on Kubernetes objects. I know what I'm looking for, and it's often not in the default table view that kubectl prints. I integrated JQ into a kubectl extension, to save you "-o json | jq" and having to pick apart the v1.List that kubectl marshals objects into. https://github.com/jrockway/kubectl-jq. That one actually has documentation, but there is a fatal flaw -- it doesn't integrate with kubectl tab completion (limitation of k8s.io/cli-runtime), so it's not too good unless you already have a target in mind, or you're targeting everything of a particular resource type. This afternoon I wanted to see the image tag of every pod that wasn't terminated (some old Job runs exist in the namespace), and that's easy to do with JQ: `kubectl jq pods 'select(.status.containerStatuses[].state.terminated == null) | .spec.containers[].image'`. I have no idea how you'd do such a thing without JQ, probably just `kubectl describe pods | grep something` and do the filtering in your head. (The recipes in the kubectl-jq documentation are pretty useful. One time I had a Kubernetes secret that had a key set to a (base64-encoded) JSON file containing a base64-encoded piece of data I wanted. Easy to fix with jq; `.data.THING | @base64d | fromjson | .actualValue | @base64d`.

JQ is something I definitely can't live without. But I will admit to sometimes preprocessing the input with grep, `select(.key|test("regex"))` is awfully verbose compared to "grep regex" ;)

• kubectl-jq

  1 12 2.6 Go

  Kubectl plugin that works like "kubectl get" but runs everything through a JQ program you provide

Big fan of JQ. I like it more than the traditional UNIX suite of text manipulation commands, because I get closer to "querying" rather than just filtering. It has really made me rethink where I want "interacting with a computer" to go in the future -- less typing commands, more querying stuff.

I have a few utilities involving JQ that I wrote.

For structured logs, I have jlog. Pipe JSON structured logs into it, and it pretty-prints the logs. For example, time zones are converted to your local time, if you choose; or you can make the timestamps relative to each other, or now. It includes jq so that you can select relevant log lines, delete spammy fields, join fields together, etc. Basically, every time you run it, you get the logs YOU want to look at. https://github.com/jrockway/json-logs. Not to oversell it, but this is one of the few pieces of software I've written that passes the toothbrush test -- I use it twice a day, every day. All the documentation is in --help; I should really paste that into the Github readme.

I am also a big fan of using JQ on Kubernetes objects. I know what I'm looking for, and it's often not in the default table view that kubectl prints. I integrated JQ into a kubectl extension, to save you "-o json | jq" and having to pick apart the v1.List that kubectl marshals objects into. https://github.com/jrockway/kubectl-jq. That one actually has documentation, but there is a fatal flaw -- it doesn't integrate with kubectl tab completion (limitation of k8s.io/cli-runtime), so it's not too good unless you already have a target in mind, or you're targeting everything of a particular resource type. This afternoon I wanted to see the image tag of every pod that wasn't terminated (some old Job runs exist in the namespace), and that's easy to do with JQ: `kubectl jq pods 'select(.status.containerStatuses[].state.terminated == null) | .spec.containers[].image'`. I have no idea how you'd do such a thing without JQ, probably just `kubectl describe pods | grep something` and do the filtering in your head. (The recipes in the kubectl-jq documentation are pretty useful. One time I had a Kubernetes secret that had a key set to a (base64-encoded) JSON file containing a base64-encoded piece of data I wanted. Easy to fix with jq; `.data.THING | @base64d | fromjson | .actualValue | @base64d`.

JQ is something I definitely can't live without. But I will admit to sometimes preprocessing the input with grep, `select(.key|test("regex"))` is awfully verbose compared to "grep regex" ;)

• miller

  63 8,553 9.1 Go

  Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

If you work with JSON and CSV data, I'd also recommend checking out Miller: https://github.com/johnkerl/miller

From the project's description: Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

• dasel

  44 4,856 8.2 Go

  Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

For anyone wrangling with data like I do. I use https://github.com/TomWright/dasel quite a lot (it supports various formats and conversion between them) Also csvkit https://csvkit.readthedocs.io for CSV to sql. And ofcourse pandas for analysis.

• Puts Debuggerer

  3,146 91 5.5 Ruby

  Ruby library for improved puts debugging, automatically displaying bonus useful information such as source line number and source code.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• howto

  1 38 0.0

  Documenting useful things, lest I forget, and sharing is caring

If there's an API for working with data that you know well already, it's somewhat pointless to learn the (quite esoteric) jq one.

Just pipe curl to node (https://github.com/jareware/howto/blob/master/Replacing%20jq...) or Python or Ruby or whatever you already know!

• wsjq

  2 13 6.8 jq

  Whitespace interpreter and debugger in jq

jq is unsurprisingly Turing complete, so I wrote a Whitespace interpreter[0] in jq. It is able to handle real-time I/O by requesting lines on-demand from stdin, which is the main input source, with `input` and outputting strings in a stream.

With a relatively large jq program like that, it is critical that the main recursive loop run efficiently, so it's annoying that there's no way to detect whether tail call optimization was applied, other than benchmarking. It would also be nice if object values were lazily evaluated so that it would be possible to create ad hoc switches.

[0]: https://github.com/andrewarchi/wsjq

• fx

  50 18,490 9.2 Go

  Terminal JSON viewer & processor

https://github.com/antonmedv/fx which uses JavaScript as the query language.

The author of the tool has also written a guide and a screencast

- https://medium.com/@antonmedv/discover-how-to-use-fx-effecti...

- https://youtu.be/ktfeRxKog98

Suggest a related project