q VS lnav

CSV utilities (still haven't pick a favorite one...): https://github.com/harelba/q https://github.com/BurntSushi/xsv https://github.com/wireservice/csvkit https://github.com/johnkerl/miller

I think they're talking about https://github.com/harelba/q, which is not very fast.

Apparently, there were many who thought the same thing; Tools to execute SQL against CSV were trdsql, q, csvq, TextQL. They were highly functional, hoewver, had many options and no input completion. I found it just a little difficult to use.

Hi, author of q here.

Regarding the error you got, q currently does not autodetect headers, so you'd need to add -H as a flag in order to use the "country" column name. You're absolutely correct on failing-fast here - It's a bug which i'll fix.

In general regarding speed - q supports automatic caching of the CSV files (through the "-C readwrite" flag). Once it's activated, it will write the data into another file (with a .qsql extension), and will use it automatically in further queries in order to speed things considerably.

Effectively, the .qsql files are regular sqlite3 files (with some metadata), and q can be used to query them directly (or any regular sqlite3 file), including the ability to seamlessly join between multiple sqlite3 files.

http://harelba.github.io/q/#auto-caching-examples

See https://lnav.org for a powerful mini-ETL CLI power tool; it embeds SQLite, supports ~every format, has great UX and easily handles a few million rows at a time.

The code base seems like a good reference as a small Python project.

My fav option in this class of apps: https://lnav.org/ It lets you use journalctl with pipes as requested here: https://github.com/Textualize/toolong/issues/4

For local development, I cannot recommend lnav[1] enough. Discovering this tool was a game changer in my day to day life. Adding comments, filtering in/out, prettify and analyse distribution is hard to live without now.

I don't think a browser tool would fit in my workflow. I need to pipe the output to the tool.

[1] https://lnav.org/

It looks like they run an SSH server inside a Docker container defined by this Dockerfile [1]. This uses the ForceCommand directive in the sshd_config file to ensure that a specific command is run when a user connects (rather than the user connecting directly to a shell).

Depending on whether the user connects as the `playground` or `tutorial1` user they interact with a bash script that is either [2] or [3].

[1]: https://github.com/tstack/lnav/blob/master/demo/Dockerfile

[2]: https://github.com/tstack/lnav/blob/master/docs/tutorials/pl...

[3]: https://github.com/tstack/lnav/blob/master/docs/tutorials/tu...