rhizome VS sqlparse

There was actually an existing, similar in concept, Ruby JIT, Rhyzome: https://github.com/chrisseaton/rhizome.

It's good to see (IMO) the development of a JIT in a strict sense. I'm personally skeptical about the current approach (invoking a compiler separately).

On the other hand, it's important to know that JITs may take a long development time to be performant, and that they also complicate the performance profile of a virtual machine.

Lots of compiler tutorials are like this - there's very little out there to explain how compilers really work.

This is my effort - trying to show genuine data structures and processes.

https://github.com/chrisseaton/rhizome

Suggest checking out the sqlparse library for a way to do the different flavours without needing to address each case directly: https://github.com/andialbrecht/sqlparse

Gotcha, since we haven't actually written all of this yet I don't have any useful code snippets to share but we've discussed tackling the problem internally using something like sqlparse. You'd need to identify the relevant sql chunks, parse them for table dependency information and then create the relevant entities in whichever data lineage tool you were using.

sqlparse – A non-validating SQL parser module for Python

Regular expressions is a popular approach to extract information from SQL statements. However, regular expressions quickly become too complex to handle common features like WITH, sub-queries, windows clauses, aliases and quotes. sqlparse is a popular python package that uses regular expressions to parse SQL.

This one is not bad: https://github.com/andialbrecht/sqlparse.

We used this repo for this: https://github.com/andialbrecht/sqlparse. I may have miscommunicated. We didn't write the parser from scratch, we created a way for the parser to detect downstream and upstream dependencies of the resource.