Zq: An Easier (and Faster) Alternative to Jq

• zed

  13 1,306 9.4 Go

  A novel data lake based on super-structured data (by brimdata)

  

Hi, all. Author here. Thanks for all the great feedback.

I've learned a lot from your comments and pointers.

The Zed project is broader than "a jq alternative" and my bad for trying out this initial positioning. I do know there are a lot of people out there who find jq really confusing, but it's clear if you become an expert, my arguments don't hold water.

We've had great feedback from many of our users who are really productive with the blend of search, analytics, and data discovery in the Zed language, and who find manipulating eclectic data in the ZNG format to be really easy.

Anyway, we'll write more about these other aspects of the Zed project in the coming weeks and months, and in the meantime, if you find any of this intriguing and want to kick the tires, feel free to hop on our slack with questions/feedback or file GitHub issues if you have ideas for improvements or find bugs.

Thanks a million!

https://github.com/brimdata/zed

• jq

  306 25,063 0.0 C

  Discontinued Command-line JSON processor [Moved to: https://github.com/jqlang/jq] (by stedolan)

  

I completely agree when your goal is GSD just use the tools you have.

When you have time to sharpen the saw come back and dig into the details of how jq and tools like it work and where their limits are. Looking at the jq builtins[1] can be very enlightening

If you get to the point where your goal is to increase your jq skills I'd recommend looking at the jq questions on Stack Overflow and posting your own solution. Contributing a solution to https://rosettacode.org/wiki/Category:Jq is also good.

1- https://github.com/stedolan/jq/blob/master/src/builtin.jq

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• gron

  64 13,520 0.0 Go

  Make JSON greppable!

Same boat here. I ended up finding gron https://github.com/tomnomnom/gron which resolved that issue for me. Now I don't have to look up how to use jq each time I want to quickly find something in some JSON.

• brackit

  21 46 6.9 Java

  Query processor with proven optimizations, ready to use for your JSON store to query semi-structured data with JSONiq. Can also be used as an ad-hoc in-memory query processor.

  

That's one of the main steps forward for Brackit, a retargetable JSONiq query engine/compiler (http://brackit.io) and the append-only data store SirixDB (https://sirix.io) and a new web frontend. My vision is not only to explore the most recent revision but also any other older revisions, to display the diffs, to display thd results of time travel queries... help is highly welcome as I'm myself a backend engineer and working on the query engine and the data store itself :-)

• fx

  50 18,509 9.1 Go

  Terminal JSON viewer & processor

I really like fx (https://github.com/antonmedv/fx) for interactive stuff. It does exactly what I think you want. You can expand individual fields and explore the schema.

However, I really do like jq for queries and scripting, so I keep both around.

• dsq

  20 3,638 4.3 Go

  Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.

  

• q

  46 10,126 2.1 Python

  q - Run SQL directly on delimited files and multi-file sqlite databases (by harelba)

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• textql

  15 9,030 3.7 Go

  Execute SQL against structured text like CSV or TSV

• jp

  6 718 1.1 Python

  Command line interface to JMESPath - http://jmespath.org

  

• gojq

  31 3,093 8.6 Go

  Pure Go implementation of jq

jq is awesome, last time I used it is... today :)

Or rather the pure Go rewrite https://github.com/itchyny/gojq which is a better faster implementation, with bugs fixed

• rb

  3 1,488 0.0 Ruby

  Turns Ruby into a versatile command line utility

I like using `jq` to create line-delimited JSON and then using a language I know well (Node) to process it after that point. I find `jq '.[] | select(.location=="Stockholm")'` less readable than something like `nq --filter '({location}) => location === "Stockholm"'` because I'm much more used to Node syntax.

- https://github.com/thisredone/rb is a widely used ruby version of this idea

- https://github.com/KelWill/nq#readme is something similar that I wrote for my own use

• nq

  1 1 10.0 JavaScript

  sed "s/jq .key/nq '({key}) => key'/" (by KelWill)

I like using `jq` to create line-delimited JSON and then using a language I know well (Node) to process it after that point. I find `jq '.[] | select(.location=="Stockholm")'` less readable than something like `nq --filter '({location}) => location === "Stockholm"'` because I'm much more used to Node syntax.

- https://github.com/thisredone/rb is a widely used ruby version of this idea

- https://github.com/KelWill/nq#readme is something similar that I wrote for my own use

• dasel

  44 4,877 8.1 Go

  Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

I would definitely add dasel to that list. It's become my de facto serialized data converter, and regularly use it to convert between csv, toml, yaml, json, and xml using jq-ish syntaxes.

https://github.com/tomwright/dasel

• jsonparser

  15 5,355 0.0 Go

  One of the fastest alternative JSON parser for Go that does not require schema

`jj` is a little tool I wrote that uses https://github.com/buger/jsonparser

• jid

  15 6,803 2.2 Go

  json incremental digger

It took me a while to grok jq, but now that I do I kinda like it? I don't think I want to learn yet another thing.

I do like tools that complement/supplement jq though, like jid: https://github.com/simeji/jid

• fzf

  407 59,920 9.6 Go

  :cherry_blossom: A command-line fuzzy finder

Please check FZF [1] and it’s integration with ctrl-r. It’s a huge productivity boost and I cannot live without it.

[1] https://github.com/junegunn/fzf

• feedback

  89 2,881 6.4

  Discontinued Public feedback discussions for: GitHub for Mobile, GitHub Discussions, GitHub Codespaces, GitHub Sponsors, GitHub Issues and more! [Moved to: https://github.com/github-community/community]

  

There's more details here: https://github.com/github/feedback/discussions/4992#discussi...

> we have also introduced an x-robots-tag: none in the http response header of Wiki pages

> Abusive behavior in Wikis had a negative impact on our search engine ranking

• wsjq

  2 13 6.8 jq

  Whitespace interpreter and debugger in jq

• yq

  66 10,840 9.2 Go

  yq is a portable command-line YAML, JSON, XML, CSV, TOML and properties processor

yq uses jq like syntax but works with YAML, JSON and XML.

https://github.com/mikefarah/yq

• mcfly

  49 6,567 7.3 Rust

  Fly through your shell history. Great Scott!

• zbpf

  1 4 0.0 Shell
  

Wow, thanks.

Coincidentally, after hearing of a friend's woes dealing with massive amounts of CSV coming from a BPF-instrumental kernel, I played around a bit with integrating Zed and BPF. Just an experimental toy (and the repo is already out of date)...

https://github.com/brimdata/zbpf

The nice thing about Zed here is any value can be a group-by key so it's easy, for example, to use kernel stacks (an array of strings) in a grouping aggregate.

(p.s. for the record, the only thing I have to do with the modern linux BPF system is the tiny vestige of origin story it shares with the original work I did in the BSD kernel around 1990)

• sfink-tools

  2 14 6.2 HTML

  Steve Fink's random development tools

The thing that I find myself wanting, which is lacking in both jq and zq afaik, is interactive exploration. I want to move around in a large JSON file, narrow my context to the portion I'm interested in, and do specialized queries and transformations on just the data I care about.

I wrote a tool to do this -- https://github.com/hotsphink/sfink-tools/blob/master/bin/jso... -- but I do not recommend it to anyone other than as perhaps a source of inspiration. It's slow and buggy, the syntax is cryptic and just matches whatever I came up with when I had a new need, etc. It probably wouldn't exist if I had heard of jq sooner.

But for what it does, it's awesome. I can do things like:

  % json somefile.json

• sirix

  44 1,085 9.1 Java

  SirixDB is an an embeddable, bitemporal, append-only database system and event store, storing immutable lightweight snapshots. It keeps the full history of each resource. Every commit stores a space-efficient snapshot through structural sharing. It is log-structured and never overwrites data. SirixDB uses a novel page-level versioning approach.

  

That's one of the main steps forward for Brackit, a retargetable JSONiq query engine/compiler (http://brackit.io) and the append-only data store SirixDB (https://sirix.io) and a new web frontend. My vision is not only to explore the most recent revision but also any other older revisions, to display the diffs, to display thd results of time travel queries... help is highly welcome as I'm myself a backend engineer and working on the query engine and the data store itself :-)

• refactorio

  1 19 10.0 Haskell

  Refactorio is a tool with which you build and maintain software. (by SuperpowersCorp)

  

My hope was to one day add JS eval support to https://github.com/SuperpowersCorp/refactorio but as you can tell by the timestamps I haven't found any time to work on it in the last 4 years.

• jless

  34 4,499 6.3 Rust

  jless is a command-line JSON viewer designed for reading, exploring, and searching through JSON data.

You could take a look at jless [1], it allows interactive selections/browsing in JSON documents.

[1] https://jless.io/

• ramda-cli

  2 571 0.0 LiveScript

  :ram: A CLI tool for processing data with functional pipelines

Not quite that, but ramda-cli[1] which I've created solves this problem, at least for me, by offering the familiar set of functions from Ramda, and you can create pipelines with those to do operations on your data.

[1]: https://github.com/raine/ramda-cli

• nushell

  214 30,081 9.9 Rust

  A new type of shell

  

Have you checked out https://www.nushell.sh/? It seems like exactly what you're describing. Although I know of people who are happily using it as their main shell, I only really use it when I need to read and manipulate data in files.

• cheshire

  2 1,469 5.6 Clojure

  Clojure JSON and JSON SMILE (binary json format) encoding/decoding

For me, transforming JSON on the command line was a pain, another DSL to learn. Now, I can just use Babashka/ Clojure + one or two functions from Cheshire https://github.com/dakrone/cheshire where I need to. If I needed a standalone tool, I would perhaps reach for https://github.com/borkdude/jet by the same author, Michiel Borkent, as Babashka or use jq that everybody else would find more familiar.

• jet

  10 654 5.6 Clojure

  CLI to transform between JSON, EDN, YAML and Transit using Clojure (by borkdude)

For me, transforming JSON on the command line was a pain, another DSL to learn. Now, I can just use Babashka/ Clojure + one or two functions from Cheshire https://github.com/dakrone/cheshire where I need to. If I needed a standalone tool, I would perhaps reach for https://github.com/borkdude/jet by the same author, Michiel Borkent, as Babashka or use jq that everybody else would find more familiar.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project