json_profile VS fq

Not sure if this is a common problem, but I built a tool to help me quickly understand the main schema, and where most of the data is, for a new JSON file given to me. It makes the assumption that sometimes peer elements in a list will have the same structure (eg they'll be objects with similar sets of keys). If that's true, it learns the structure of the file, prints out the heaviest _aggregated_ path (meaning it thinks in terms of a directory-like structure), as well as giving you various size-per-path hints to help introduce yourself to the JSON file:

https://github.com/tylerneylon/json_profile

I see, thanks for replying and no worries! yeap some of the "self-describing" formats like msgpack, cbor etc will because of how fq works have to be decoded into something more of a meta-msgpack etc.

About blobs, if you want to change how (possibly large) binaries are represented as JSON you can use the bits_format options, see https://github.com/wader/fq/blob/master/doc/usage.md#options, so fq -o bits_format=md5 torepr ...

I can highly recommend to learn jq, it's what makes fq really useful, and as a bonus you will learn jq in general! :)

Hey! fq author here. I have a bunch of related tools in the readme https://github.com/wader/fq?tab=readme-ov-file#tools two suggestions: gnu poke and wireshark (can decode lots of more things then just network protocol)

Did someone say let's represent structured data as json? a bit of shameless plug: https://github.com/wader/fq :) It's using a fork of gojq btw!

https://github.com/wader/fq has a REPL and can read JSON. Tip is to use "paste | from_json | repl" in a REPl to paste JSON into a sub-REPL, you can also use `` with fq which is a raw string literal

I do lots of exploratory work in various structure data, in my case often debugging media filea via https://github.com/wader/fq, which mean doing lots of use-once-queries on the command line or REPL. In those cases jq line-friendly and composable syntax and generators really shine.