csvquote VS Awk-Batteries

There is a small program I wrote called csvquote[1] that can be used to sanitize input to awk so it can rely on delimiter characters (commas) to always mean delimiters. The results from awk then get piped through the same program at the end to restore the commas inside the field values.

Also works for other text processing tools like cut, sed, sort, etc.

[1] https://github.com/dbro/csvquote

CSVs with quoted fields and imbedded newlines can be troublesome in awk. Years ago I had found a script that worked for me, I'm not sure but I think it was this:

http://lorance.freeshell.org/csv/

There's also https://github.com/dbro/csvquote which is more unix-like in philosophy: it only handles transforming the CVS data into something that awk (or other utilities) can more easily deal with. I haven't used it but will probably try it next time I need something like that.

It's nice that everyone is supporting this, I've written a portable awk module that takes control of the parsing and it is SLOW (and a little buggy). I'm a little bummed that nobody will use it but this is truly a step in the right direction.

I guess for the people that are still using nawk, you can set up an AWK envvar so you can { awk -f $AWKU/ucsv.awk -f <(echo '{print NR, $1}') }

https://github.com/Nomarian/Awk-Batteries/blob/master/Units/...

if you have gawkextlib, you can -i csv, you can also download this and put it in your $AWKPATH then you just -i ucsv and use the csv file as normal. if you need an array with headers let me know.

I wrote a parser that does what -F, does but correctly, you can see that its actually very difficult. Not only that, but there are extensions to csv, some csv have a header which means that instead of $1 $2 $3 ..., you just name the field instead, which means you have to ignore the first record. there's also other things, but csv is a difficult format to parse.

My attempt in awk

awk lua