qsv VS tsv-utils

Thanks for the detailed feedback @snidane!

As maintainer of qsv, here's my reply:

- Given qsv's rapid release cycle (173 releases over three years), the auto-update check is essential at the moment. Once we reach 1.0, I'll turn it off. For now, given your feedback, I've only made it check 10% of the time.

- Pivot is in the backlog and I'll be sure to add unpivot when I implement it. (https://github.com/jqnatividad/qsv/issues/799)

- I'll add a dedicated summing command with the group by (-by) and window by (-over) capability (https://github.com/jqnatividad/qsv/issues/1514). Do note that `stats` has basic sum as @ezequiel-garzon pointed out.

- With the `enum` command, qsv can achieve what you proposed with `laminate`. E.g. qsv enum --new-column newcol --constant newconstant mydata.csv --output laminated-data.csv

- With the cat rowskey command, qsv can already concatenate files with mismatched headers.

- other file formats. qsv supports parquet, csv, tsv, excel, ods, datapackage, sqlite and more (see https://github.com/jqnatividad/qsv/tree/master#file-formats). Fixed-format though is not supported yet and quite interesting, and have added it to the backlog (https://github.com/jqnatividad/qsv/issues/1515)

- as to "enable embedding outputs of commands", qsv is composable by design, so you can use standard stdin/stdout redirection/piping techniques to have it work with other CLI tools like jq, awk, etc.

Finally, just released v0.120.0 that already incorporates the less aggressive self-update check. https://github.com/jqnatividad/qsv/releases/tag/0.120.0

qsv [1] also has a sqlp command which lets you run Polars SQL queries (even on multiple files). Here I'll send the csv data from stdin (represented by -) and then (optionally) pipe the output to the table command for formatting. The shape of the result is also printed to stderr (the (4, 2) below).

[1] https://github.com/jqnatividad/qsv

$ echo 'Name,Department,Salary

It works quite well IMHO. Using the mlua crate, I’ve managed to integrate Luau as a very powerful data-wrangling DSL for qsv (https://github.com/jqnatividad/qsv)

Maybe this might be better done using this? https://github.com/jqnatividad/qsv

https://github.com/jqnatividad/qsv is another option.

If you need just csv/tsv parsing, you can also take a look at https://github.com/eBay/tsv-utils

You might want to look at tsv-utils, or a similar project: https://github.com/eBay/tsv-utils

For the SQL part, but maybe a lot heavier, you can use one of the projects listed on this page: https://github.com/multiprocessio/dsq (No longer maintained, but has links to lots of other projects)

TSV is most often a better format than CSV. Localization, in particular, is a nightmare with CSV.

For anything down and dirty, what's wrong with -F'"'? For anything fancy there are plenty of things like the below.

eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.

includes csv to tsv: https://github.com/eBay/tsv-utils

HT: https://simonwillison.net/

As an example, eBay's tsv-utils took full advantage of the GC and performed better than existing programs that had been hand-optimized in C etc.

tsv-utils - Command line csv data manipulation toolkit. D

In the majority of cases you'll be better off with Tab Separated Values over Comma Separated Values. More info here.

May well be overkill for your needs, but I'm a fan of tsv-utils It's fast and enormously flexible, and seems to me a "best of breed" toolset for data mining CSV files (that is what it was written for). https://github.com/eBay/tsv-utils