csvq VS visidata

sure can do, if you already use that shell [1], but personally I like specific tools for specific jobs such as jq [2], fx, csvq [3] etc, there's value in decoupling shells from utils (modularity, speed, innovation etc).

[1] I don't but tempted to try, like its data-types concept

[2] https://jqlang.github.io/jq/

[3] https://github.com/mithrandie/csvq

csvq

There is a way of running SQL-like queries against CSV files.

Lately I have had to do a lot of flat file analysis and tools along these lines have been a godsend. Will check this out.

My go to lately has been csvq (https://mithrandie.github.io/csvq/). Really nice to be able run complicated selects right over a CSV file with no setup at all.

csvq (https://mithrandie.github.io/csvq/)

I usually do this with awk, my largest target files being half a TB in size for a project last year (and far too large to hold entirely in RAM). There are some other utilities like csvq and csvsql both of which let you write SQL-style queries against CSV files, but I'm not sure how they perform on large files. There's a nice list of CSV manipulation tools too if any of those jog your memory.

Apparently, there were many who thought the same thing; Tools to execute SQL against CSV were trdsql, q, csvq, TextQL. They were highly functional, hoewver, had many options and no input completion. I found it just a little difficult to use.

Please check out this tool https://github.com/mithrandie/csvq

If you are looking to query nondb data with sql statements then you may want to check something like https://github.com/mithrandie/csvq (SQL for csv).

I'd like to call out one of my favorite pieces of software from the past 10 years: VisiData [1] has completely changed the way I do ad-hoc data processing, and is now my go-to for pretty much all use cases that I previously used spreadsheets for, and about half of those I previously used databases for.

It's a TUI application, not strictly CLI, but scriptable, and I figure anyone building pipelines using tools like jq, q, awk, grep, etc. to process tabular data will find it extremely useful.

----

[1]: https://visidata.org

[4] "Is it possible to "flatten" structured data (like JSON?)": https://github.com/saulpw/visidata/discussions/1605

Visidata - https://visidata.org

You can try Visidata

VisiData is an awesome terminal spreadsheet tool. edbrowse for internet browsing.

Have you ever tried out visidata? It's not vim, but it's a terminal app with vim-like keybindings for visualizing tabular data (and it can convert from other types like json). Not quite a neovim buffer, but you could always open visidata in a new terminal buffer.

If you are a command-line user, try visidata[0]

[0] https://github.com/saulpw/visidata

The datasets will be available as SQLite, JSONL, and CSV. This will be great for sharpening your SQL/Python/VisiData skills.

Help Sarah find the family holiday tapestry before her father notices it's missing! Sharpen your SQL/Python/VisiData skills with Hanukkah of Data.