sqlite_protobuf VS csvq

SQLite's virtual table API (https://www.sqlite.org/vtab.html) makes it possible to access other data structures through the query engine. You don't need to know much if anything about how the database engine executes queries, you only need to implement the callbacks it needs to do its job. A few years ago I wrote an extension to let me search through serialized Protobufs which were stored as blobs in a regular database.

https://github.com/rgov/sqlite_protobuf

Some time ago I joined a project that was recording logs in Protobuf format. I demonstrated that we could log the serialized Protobuf structures to a SQLite database and then use a plugin to query across the message fields: https://github.com/rgov/sqlite_protobuf

Learning how to use the SQLite extension API was a little mind-expanding but it worked like a charm.

sure can do, if you already use that shell [1], but personally I like specific tools for specific jobs such as jq [2], fx, csvq [3] etc, there's value in decoupling shells from utils (modularity, speed, innovation etc).

[1] I don't but tempted to try, like its data-types concept

[2] https://jqlang.github.io/jq/

[3] https://github.com/mithrandie/csvq

csvq

There is a way of running SQL-like queries against CSV files.

Lately I have had to do a lot of flat file analysis and tools along these lines have been a godsend. Will check this out.

My go to lately has been csvq (https://mithrandie.github.io/csvq/). Really nice to be able run complicated selects right over a CSV file with no setup at all.

csvq (https://mithrandie.github.io/csvq/)

I usually do this with awk, my largest target files being half a TB in size for a project last year (and far too large to hold entirely in RAM). There are some other utilities like csvq and csvsql both of which let you write SQL-style queries against CSV files, but I'm not sure how they perform on large files. There's a nice list of CSV manipulation tools too if any of those jog your memory.

Apparently, there were many who thought the same thing; Tools to execute SQL against CSV were trdsql, q, csvq, TextQL. They were highly functional, hoewver, had many options and no input completion. I found it just a little difficult to use.

Please check out this tool https://github.com/mithrandie/csvq

If you are looking to query nondb data with sql statements then you may want to check something like https://github.com/mithrandie/csvq (SQL for csv).