sqlparse VS zetasql

Suggest checking out the sqlparse library for a way to do the different flavours without needing to address each case directly: https://github.com/andialbrecht/sqlparse

Gotcha, since we haven't actually written all of this yet I don't have any useful code snippets to share but we've discussed tackling the problem internally using something like sqlparse. You'd need to identify the relevant sql chunks, parse them for table dependency information and then create the relevant entities in whichever data lineage tool you were using.

sqlparse – A non-validating SQL parser module for Python

Regular expressions is a popular approach to extract information from SQL statements. However, regular expressions quickly become too complex to handle common features like WITH, sub-queries, windows clauses, aliases and quotes. sqlparse is a popular python package that uses regular expressions to parse SQL.

This one is not bad: https://github.com/andialbrecht/sqlparse.

There are even table-valued functions.

These things are not widespread, and differ by implementation, and the way these are used by clients are copy-and-paste. Something as thoughtful as ZetaSQL https://github.com/google/zetasql does not have mechanisms for structuring (modules, packages, interfaces). SQL will not, cannot evolve into such a direction (or, anything that evolves, will not be recognizable as SQL).

Hi, I develop a BigQuery emulator ( https://github.com/goccy/bigquery-emulator ) from early 2022. It is written in Go, but can be used from bq command line tool and other language's (e.g. Python ) client SDK after installing docker image or released binary. It currently supports over 200 of the nearly 330 standard functions in BigQuery and all data types except GEOGRAPHY ( see https://github.com/goccy/go-zetasqlite#status for details ). ZetaSQL ( https://github.com/google/zetasql ) is used to parse and analyze queries.

If you don't want to do it yourself, there's this:

https://github.com/google/zetasql

Parsing is huge but it's amazing how small a part of the job it is. This library isn't even the half of it.

https://github.com/google/zetasql

It is amazingly good.

You give it textual SQL (+ schema + all your function definitions) and it returns a really clean logical query plan. It is also happy to do this via a protobufs so you can use it from languages other than C++. It is also tested and documented up the wazoo. It has been such a pleasure to work with.

Anyway, the big problem with ZetaSQL is that it is not a common SQL dialect.

It seems that the only reasonable way to do this PostgreSQL interface for Cloud Spanner is to add a second parser (and other extensions) to ZetaSQL. If I am correct, I really really hope they open source that part of ZetaSQL as well - it would be a massive step forward for open source SQL tooling.

I mean the postgres parser (and semantic changes) for ZetaSQL. The zetasql parser is in a file called zetasql/parser/bison_parser.y, I strongly suspect they now have a file called something like zetasql/pgparser/bison_parser.y as well (and much more pervasive changes to support the deeper differences in the dialects).

This is the lexical structure and syntax docs for the new postgres inteface to cloud spanner:

https://cloud.google.com/spanner/docs/postgresql/lexical

And this is the zetasql lexical structure and syntax docs:

https://github.com/google/zetasql/blob/master/docs/lexical.m...

Notice that the new PG docs are an edit of the Zeta ones - evidence that my hypothesis is correct.

zetasql implements BigQuery, Spanner, and Dataflow dialects.

ZetaSQL[1] seems like it could be a fit for your use case. I've worked with Apache Calcite in the past and found it to be very complex to work with. I found ZetaSQL to be a little easier to use.

[1] https://github.com/google/zetasql

I’m not aware of one but you could probably use ZetaSQL to put one together, the difficult work has been opened, you’d just need to add the LSP layer.

Some additional helper functions here.