Zq: An Easier (and Faster) Alternative to Jq

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • LearnThisRepo.com - Learn 300+ open source libraries for free using AI.
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern API for authentication & user identity.
  • zed

    A novel data lake based on super-structured data (by brimdata)

    Hi, all. Author here. Thanks for all the great feedback.

    I've learned a lot from your comments and pointers.

    The Zed project is broader than "a jq alternative" and my bad for trying out this initial positioning. I do know there are a lot of people out there who find jq really confusing, but it's clear if you become an expert, my arguments don't hold water.

    We've had great feedback from many of our users who are really productive with the blend of search, analytics, and data discovery in the Zed language, and who find manipulating eclectic data in the ZNG format to be really easy.

    Anyway, we'll write more about these other aspects of the Zed project in the coming weeks and months, and in the meantime, if you find any of this intriguing and want to kick the tires, feel free to hop on our slack with questions/feedback or file GitHub issues if you have ideas for improvements or find bugs.

    Thanks a million!

    https://github.com/brimdata/zed

  • jq

    Command-line JSON processor [Moved to: https://github.com/jqlang/jq] (by stedolan)

    I completely agree when your goal is GSD just use the tools you have.

    When you have time to sharpen the saw come back and dig into the details of how jq and tools like it work and where their limits are. Looking at the jq builtins[1] can be very enlightening

    If you get to the point where your goal is to increase your jq skills I'd recommend looking at the jq questions on Stack Overflow and posting your own solution. Contributing a solution to https://rosettacode.org/wiki/Category:Jq is also good.

    1- https://github.com/stedolan/jq/blob/master/src/builtin.jq

  • LearnThisRepo.com

    Learn 300+ open source libraries for free using AI. LearnThisRepo lets you learn 300+ open source repos including Postgres, Langchain, VS Code, and more by chatting with them using AI!

  • gron

    Make JSON greppable!

    Same boat here. I ended up finding gron https://github.com/tomnomnom/gron which resolved that issue for me. Now I don't have to look up how to use jq each time I want to quickly find something in some JSON.

  • brackit

    Query processor with proven optimizations, ready to use for your JSON store to query semi-structured data with JSONiq. Can also be used as an ad-hoc in-memory query processor.

    That's one of the main steps forward for Brackit, a retargetable JSONiq query engine/compiler (http://brackit.io) and the append-only data store SirixDB (https://sirix.io) and a new web frontend. My vision is not only to explore the most recent revision but also any other older revisions, to display the diffs, to display thd results of time travel queries... help is highly welcome as I'm myself a backend engineer and working on the query engine and the data store itself :-)

  • fx

    Terminal JSON viewer & processor

    I really like fx (https://github.com/antonmedv/fx) for interactive stuff. It does exactly what I think you want. You can expand individual fields and explore the schema.

    However, I really do like jq for queries and scripting, so I keep both around.

  • dsq

    Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.

  • q

    q - Run SQL directly on delimited files and multi-file sqlite databases (by harelba)

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • textql

    Execute SQL against structured text like CSV or TSV

  • jp

    Command line interface to JMESPath - http://jmespath.org

  • gojq

    Pure Go implementation of jq

    jq is awesome, last time I used it is... today :)

    Or rather the pure Go rewrite https://github.com/itchyny/gojq which is a better faster implementation, with bugs fixed

  • rb

    Turns Ruby into a versatile command line utility

    I like using `jq` to create line-delimited JSON and then using a language I know well (Node) to process it after that point. I find `jq '.[] | select(.location=="Stockholm")'` less readable than something like `nq --filter '({location}) => location === "Stockholm"'` because I'm much more used to Node syntax.

    - https://github.com/thisredone/rb is a widely used ruby version of this idea

    - https://github.com/KelWill/nq#readme is something similar that I wrote for my own use

  • nq

    sed "s/jq .key/nq '({key}) => key'/" (by KelWill)

    I like using `jq` to create line-delimited JSON and then using a language I know well (Node) to process it after that point. I find `jq '.[] | select(.location=="Stockholm")'` less readable than something like `nq --filter '({location}) => location === "Stockholm"'` because I'm much more used to Node syntax.

    - https://github.com/thisredone/rb is a widely used ruby version of this idea

    - https://github.com/KelWill/nq#readme is something similar that I wrote for my own use

  • dasel

    Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

    I would definitely add dasel to that list. It's become my de facto serialized data converter, and regularly use it to convert between csv, toml, yaml, json, and xml using jq-ish syntaxes.

    https://github.com/tomwright/dasel

  • jsonparser

    One of the fastest alternative JSON parser for Go that does not require schema

    `jj` is a little tool I wrote that uses https://github.com/buger/jsonparser

  • jid

    json incremental digger

    It took me a while to grok jq, but now that I do I kinda like it? I don't think I want to learn yet another thing.

    I do like tools that complement/supplement jq though, like jid: https://github.com/simeji/jid

  • fzf

    :cherry_blossom: A command-line fuzzy finder

    Please check FZF [1] and it’s integration with ctrl-r. It’s a huge productivity boost and I cannot live without it.

    [1] https://github.com/junegunn/fzf

  • feedback

    Public feedback discussions for: GitHub for Mobile, GitHub Discussions, GitHub Codespaces, GitHub Sponsors, GitHub Issues and more! [Moved to: https://github.com/github-community/community]

    There's more details here: https://github.com/github/feedback/discussions/4992#discussi...

    > we have also introduced an x-robots-tag: none in the http response header of Wiki pages

    > Abusive behavior in Wikis had a negative impact on our search engine ranking

  • wsjq

    Whitespace interpreter and debugger in jq

  • yq

    yq is a portable command-line YAML, JSON, XML, CSV, TOML and properties processor

    yq uses jq like syntax but works with YAML, JSON and XML.

    https://github.com/mikefarah/yq

  • mcfly

    Fly through your shell history. Great Scott!

  • zbpf

    Wow, thanks.

    Coincidentally, after hearing of a friend's woes dealing with massive amounts of CSV coming from a BPF-instrumental kernel, I played around a bit with integrating Zed and BPF. Just an experimental toy (and the repo is already out of date)...

    https://github.com/brimdata/zbpf

    The nice thing about Zed here is any value can be a group-by key so it's easy, for example, to use kernel stacks (an array of strings) in a grouping aggregate.

    (p.s. for the record, the only thing I have to do with the modern linux BPF system is the tiny vestige of origin story it shares with the original work I did in the BSD kernel around 1990)

  • sfink-tools

    Steve Fink's random development tools

    The thing that I find myself wanting, which is lacking in both jq and zq afaik, is interactive exploration. I want to move around in a large JSON file, narrow my context to the portion I'm interested in, and do specialized queries and transformations on just the data I care about.

    I wrote a tool to do this -- https://github.com/hotsphink/sfink-tools/blob/master/bin/jso... -- but I do not recommend it to anyone other than as perhaps a source of inspiration. It's slow and buggy, the syntax is cryptic and just matches whatever I came up with when I had a new need, etc. It probably wouldn't exist if I had heard of jq sooner.

    But for what it does, it's awesome. I can do things like:

      % json somefile.json

  • sirix

    SirixDB is an an embeddable, bitemporal, append-only database system and event store, storing immutable lightweight snapshots. It keeps the full history of each resource. Every commit stores a space-efficient snapshot through structural sharing. It is log-structured and never overwrites data. SirixDB uses a novel page-level versioning approach.

    That's one of the main steps forward for Brackit, a retargetable JSONiq query engine/compiler (http://brackit.io) and the append-only data store SirixDB (https://sirix.io) and a new web frontend. My vision is not only to explore the most recent revision but also any other older revisions, to display the diffs, to display thd results of time travel queries... help is highly welcome as I'm myself a backend engineer and working on the query engine and the data store itself :-)

  • refactorio

    Refactorio is a tool with which you build and maintain software. (by SuperpowersCorp)

    My hope was to one day add JS eval support to https://github.com/SuperpowersCorp/refactorio but as you can tell by the timestamps I haven't found any time to work on it in the last 4 years.

  • jless

    jless is a command-line JSON viewer designed for reading, exploring, and searching through JSON data.

    You could take a look at jless [1], it allows interactive selections/browsing in JSON documents.

    [1] https://jless.io/

  • ramda-cli

    :ram: A CLI tool for processing data with functional pipelines

    Not quite that, but ramda-cli[1] which I've created solves this problem, at least for me, by offering the familiar set of functions from Ramda, and you can create pipelines with those to do operations on your data.

    [1]: https://github.com/raine/ramda-cli

  • nushell

    A new type of shell

    Have you checked out https://www.nushell.sh/? It seems like exactly what you're describing. Although I know of people who are happily using it as their main shell, I only really use it when I need to read and manipulate data in files.

  • cheshire

    Clojure JSON and JSON SMILE (binary json format) encoding/decoding

    For me, transforming JSON on the command line was a pain, another DSL to learn. Now, I can just use Babashka/ Clojure + one or two functions from Cheshire https://github.com/dakrone/cheshire where I need to. If I needed a standalone tool, I would perhaps reach for https://github.com/borkdude/jet by the same author, Michiel Borkent, as Babashka or use jq that everybody else would find more familiar.

  • jet

    CLI to transform between JSON, EDN, YAML and Transit using Clojure (by borkdude)

    For me, transforming JSON on the command line was a pain, another DSL to learn. Now, I can just use Babashka/ Clojure + one or two functions from Cheshire https://github.com/dakrone/cheshire where I need to. If I needed a standalone tool, I would perhaps reach for https://github.com/borkdude/jet by the same author, Michiel Borkent, as Babashka or use jq that everybody else would find more familiar.

  • WorkOS

    The modern API for authentication & user identity. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts