parser VS sqlparse

For MySQL the situation is a bit different. sqlc uses the parser of TiDB (https://github.com/pingcap/parser), which is a parser that aims to be basically compatible with MySQL, but is quite young and is not a MySQL parser. The most basic queries work, but even simple joins or aggregations usually result in variables with unknown data types or wrong nullability. So you loose a lot of the benefits of sqlc. Manual type annotations for MySQL also do not work most of the time. They are simply ignored and forwarded to MySQL as invalid query if they do not occur on a place where sqlc is expecting them.

This is a disaster waiting to happen. Regular expressions should never be used to parse non-regular languages, of which SQL is one.

There are a variety of mature MySQL dialect parsers available[1][2], and MySQL should have its own public APIs for transforming a query into an AST. Any of those would be a safer and more correct alternative.

[1]: https://github.com/pingcap/parser

[2]: https://github.com/square/mysql-parser

Pingcap parser is a MySQL parser in Go.

Suggest checking out the sqlparse library for a way to do the different flavours without needing to address each case directly: https://github.com/andialbrecht/sqlparse

Gotcha, since we haven't actually written all of this yet I don't have any useful code snippets to share but we've discussed tackling the problem internally using something like sqlparse. You'd need to identify the relevant sql chunks, parse them for table dependency information and then create the relevant entities in whichever data lineage tool you were using.

sqlparse – A non-validating SQL parser module for Python

Regular expressions is a popular approach to extract information from SQL statements. However, regular expressions quickly become too complex to handle common features like WITH, sub-queries, windows clauses, aliases and quotes. sqlparse is a popular python package that uses regular expressions to parse SQL.

This one is not bad: https://github.com/andialbrecht/sqlparse.

We used this repo for this: https://github.com/andialbrecht/sqlparse. I may have miscommunicated. We didn't write the parser from scratch, we created a way for the parser to detect downstream and upstream dependencies of the resource.