jsonslicer VS jc

> JSON’s design assumes the user can read the entire file into memory

No? The design of most JSON libraries assumes that, but there are perfectly good incremental JSON parsers out there[1–3]. It’s just that people don’t seem to have figured out a good API for not-completely-incremental parsing (please prove me wrong here!), but this applies equally to any structured data format as soon as you want to pull out pieces of data that are nested more than one level down.

The lack of length prefixes in JSON does indeed make a solid parser somewhat more difficult, but you get the ability to author and validate it manually instead. All in all a draw and not because of the incremental parsing thing.

(Tabular or otherwise homogeneous data is indeed reprsented wastefully, but unless the individual records are huge json+gzip is a perfectly serviceable “worse-is-better” solution.)

[1] https://github.com/ICRAR/ijson

[2] https://github.com/AMDmi3/jsonslicer

[3] https://github.com/danielyule/naya

https://github.com/kellyjonbrazil/jc - "CLI tool and python library that converts the output of popular command-line tools, file-types, and common strings to JSON, YAML, or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts."

> I'd love to see programs communicate through a typed JSON/proto format that shed enough details to make this more independent, and get useful shell command structuring/completion or full blown GUIs from simply introspecting the expected input and output types.

You should try PowerShell. It's basically Microsoft's .NET ecosystem molded into an interactive command line. I'm not entirely sure if PoweShell can make full use of the static types that build up its core, but its ability to exchange objects in the command line is almost unmatched.

On Linux you can use `jc` (https://github.com/kellyjonbrazil/jc) combined with `jq` (https://jqlang.github.io/jq/) to glue together command lines.

In addition to my previous comment about jq-like tools, I want to share a couple other interesting tools, which I use alongside jq are jo [0] and jc [1].

[0]: https://github.com/jpmens/jo

[1]: https://github.com/kellyjonbrazil/jc

> I wanted to write some wrappers for the standard commands that automatically did all this via `jq`.

If you're not already aware of it, you may wish to check out `jc`[0] which describes itself as a "CLI tool and python library that converts the output of popular command-line tools, file-types, and common strings to JSON, YAML, or Dictionaries. This allows piping of output to tools like jq..."

The `jc` documentation[1] & parser[2] for `ls` also demonstrates that reliable & cross-platform parsing of even "basic" commands can be non-trivial.

[0] https://github.com/kellyjonbrazil/jc

[1] https://kellyjonbrazil.github.io/jc/docs/parsers/ls

[2] https://github.com/kellyjonbrazil/jc/blob/4cd721be8595db52b6...