elk VS sqlparse

Graphviz is the classic option but unfortunately it isn't very good. I mean it was great when it was written in the 80s or whatever but then it seems like it was declared "done" and is still stuck in the 80s.

Quite annoying because it totally dominates the mindshare of graph layout tools, making it difficult to find alternatives.

Here's some other options anyway:

* Eclipse Layout Kernel: https://github.com/eclipse/elk

* OGDF: https://ogdf.uos.de/

In fairness both their websites are pretty terrible (would some examples kill you OGDF?) and they don't provide an easy way to try them out, so I guess it's not that surprising that Graphviz dominates.

Anyway in practice if you have a complex graph then doing it manually is by far the best option.

If it's too big to do manually then it's unlikely to be a useful graph in the first place.

The issue I have with a lot of these tools is they work fine when depicting relationships between tables in the same schema (talking mainly about PostgreSQL databases), but few support showing relationships between tables across different schemas.

Also, when the number of tables grows large, few have layouts arranged in an optimal way. I use D2 (https://d2lang.com/) to create ERDs. However, of the free layout engines available in D2, Dagre (https://github.com/dagrejs/dagre) and ELK (https://github.com/eclipse/elk) both don't have optimal placement of layouts for a sufficiently complicated database.

Suggest checking out the sqlparse library for a way to do the different flavours without needing to address each case directly: https://github.com/andialbrecht/sqlparse

Gotcha, since we haven't actually written all of this yet I don't have any useful code snippets to share but we've discussed tackling the problem internally using something like sqlparse. You'd need to identify the relevant sql chunks, parse them for table dependency information and then create the relevant entities in whichever data lineage tool you were using.

sqlparse – A non-validating SQL parser module for Python

Regular expressions is a popular approach to extract information from SQL statements. However, regular expressions quickly become too complex to handle common features like WITH, sub-queries, windows clauses, aliases and quotes. sqlparse is a popular python package that uses regular expressions to parse SQL.

This one is not bad: https://github.com/andialbrecht/sqlparse.

We used this repo for this: https://github.com/andialbrecht/sqlparse. I may have miscommunicated. We didn't write the parser from scratch, we created a way for the parser to detect downstream and upstream dependencies of the resource.