csvq VS yq

sure can do, if you already use that shell [1], but personally I like specific tools for specific jobs such as jq [2], fx, csvq [3] etc, there's value in decoupling shells from utils (modularity, speed, innovation etc).

[1] I don't but tempted to try, like its data-types concept

[2] https://jqlang.github.io/jq/

[3] https://github.com/mithrandie/csvq

csvq

There is a way of running SQL-like queries against CSV files.

Lately I have had to do a lot of flat file analysis and tools along these lines have been a godsend. Will check this out.

My go to lately has been csvq (https://mithrandie.github.io/csvq/). Really nice to be able run complicated selects right over a CSV file with no setup at all.

csvq (https://mithrandie.github.io/csvq/)

I usually do this with awk, my largest target files being half a TB in size for a project last year (and far too large to hold entirely in RAM). There are some other utilities like csvq and csvsql both of which let you write SQL-style queries against CSV files, but I'm not sure how they perform on large files. There's a nice list of CSV manipulation tools too if any of those jog your memory.

Apparently, there were many who thought the same thing; Tools to execute SQL against CSV were trdsql, q, csvq, TextQL. They were highly functional, hoewver, had many options and no input completion. I found it just a little difficult to use.

Please check out this tool https://github.com/mithrandie/csvq

If you are looking to query nondb data with sql statements then you may want to check something like https://github.com/mithrandie/csvq (SQL for csv).

Install jq and yq, and wrap your commands with | yq -y ..

Confusingly there is another project called yq that does exactly what you're suggesting and it's a preprocessor that converts yaml to json and then used jq. https://github.com/kislyuk/yq

yq

I personally find the yq tool from https://github.com/kislyuk/yq much more useful: it has all the same options and formats as `jq` (as it's really a wrapper around jq). Rather than the `yq` in the OP here where only partial functionality exists.

XML data from the API can be parsed and filtered with xq. There may be multiple ways to get it; first try the yq toolset which includes it.

There is also yq [1], which attempts the same for yaml, toml and xml. (And confusingly also contains a binary named "xq" for querying xml, however with a different syntax)

[1] https://github.com/kislyuk/yq