Modernizing AWK, a 45-year old language, by adding CSV support

1. tsv-utils

   1 10 1,446 0.0 D

   eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.

   

   For anything down and dirty, what's wrong with -F'"'? For anything fancy there are plenty of things like the below.

   eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.

   includes csv to tsv: https://github.com/eBay/tsv-utils

   HT: https://simonwillison.net/

2. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

3. usv

   2 19 215 6.0 Shell

   Unicode Separated Values (USV) data markup for units, records, groups, files, streaming, and more.

   

   Ben this is great, thank you. Would you be open to adding Unicode Separated Values (USV) as well? It's much like CSV and also simpler because of no escaping and no quoting. I can donate $50 to you or your charity of choice as a token of thanks and encouragement.

   https://github.com/sixarm/usv

4. nio

   3 7 35 7.1 Nim

   Low Overhead Numerical/Native IO library & tools (by c-blake)

   

5. miller

   4 67 9,348 9.0 Go

   Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

   

   I have grown fond of using miller[0] to handle command line data processing. Handles the standard tabular formats (csv, tsv, json) and has all of the standard data cleanup options. Works on streams so (most operations) are not limited by memory.

   [0]: https://github.com/johnkerl/miller

6. simonwillisonblog

   5 43 305 8.6 Python

   The source code behind my blog

   

   For anything down and dirty, what's wrong with -F'"'? For anything fancy there are plenty of things like the below.

   eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.

   includes csv to tsv: https://github.com/eBay/tsv-utils

   HT: https://simonwillison.net/

7. xsv

   6 67 10,665 0.0 Rust

   Discontinued A fast CSV command line toolkit written in Rust.

   

   xsv does similar stuff for CSV, and very rapidly: https://github.com/BurntSushi/xsv

   https://miller.readthedocs.io/en/latest/why/ has a nice section on "why miller":

   > First: there are tools like xsv which handles CSV marvelously and jq which handles JSON marvelously, and so on -- but I over the years of my career in the software industry I've found myself, and others, doing a lot of ad-hoc things which really were fundamentally the same except for format. So the number one thing about Miller is doing common things while supporting multiple formats: (a) ingest a list of records where a record is a list of key-value pairs (however represented in the input files); (b) transform that stream of records; (c) emit the transformed stream -- either in the same format as input, or in a different format.

8. csvquote

   7 2 33 0.0 Go

   

9. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

10. qsv

    8 18 2,969 10.0 Rust

    Blazing-fast Data-Wrangling toolkit

    

    I was using xsv a lot at work (it is so much faster than csvkit) but I've recently jumped to qsv, a fork with more features.

    https://github.com/jqnatividad/qsv

11. zsv

    9 27 230 9.1 C

    zsv+lib: tabular data swiss-army knife CLI + world's fastest (simd) CSV parser

    

    This is not an outlier. `mlr` is quite slow, literally off-the-charts slow for our purposes when we benchmarked vs xsv and zsv (see https://github.com/liquidaty/zsv. disclaimer: I'm one of its authors)

12. goawk

    10 20 1,986 6.7 Go

    A POSIX-compliant AWK interpreter written in Go, with CSV support

    

    I've also added explicit tests for ASCII and Unicode unit and record separators, just to ensure I don't regress: https://github.com/benhoyt/goawk/commit/215652de58f33630edb0...

Suggest a related project