sqlparse VS dagre

Suggest checking out the sqlparse library for a way to do the different flavours without needing to address each case directly: https://github.com/andialbrecht/sqlparse

Gotcha, since we haven't actually written all of this yet I don't have any useful code snippets to share but we've discussed tackling the problem internally using something like sqlparse. You'd need to identify the relevant sql chunks, parse them for table dependency information and then create the relevant entities in whichever data lineage tool you were using.

sqlparse – A non-validating SQL parser module for Python

Regular expressions is a popular approach to extract information from SQL statements. However, regular expressions quickly become too complex to handle common features like WITH, sub-queries, windows clauses, aliases and quotes. sqlparse is a popular python package that uses regular expressions to parse SQL.

This one is not bad: https://github.com/andialbrecht/sqlparse.

We used this repo for this: https://github.com/andialbrecht/sqlparse. I may have miscommunicated. We didn't write the parser from scratch, we created a way for the parser to detect downstream and upstream dependencies of the resource.

It was unmaintained for a long time (maybe considered done implementing the algorithm it set out to), but recently a Google engineer has revived it: https://github.com/dagrejs/dagre

The issue I have with a lot of these tools is they work fine when depicting relationships between tables in the same schema (talking mainly about PostgreSQL databases), but few support showing relationships between tables across different schemas.

Also, when the number of tables grows large, few have layouts arranged in an optimal way. I use D2 (https://d2lang.com/) to create ERDs. However, of the free layout engines available in D2, Dagre (https://github.com/dagrejs/dagre) and ELK (https://github.com/eclipse/elk) both don't have optimal placement of layouts for a sufficiently complicated database.

You are right, after a bit of searching found nice library: https://github.com/dagrejs/dagre and couple of others. Maybe will try it out.

I am using ReactFlow with Dagre to create reasonably simple graph flows as the following example:

Hello,

I've been researching layered graph algorithms, but it appears that this market is mostly inactive at the moment.

Dagre (https://github.com/dagrejs/dagre) is considered outdated, with a codebase from 2015 that is no longer being updated, although they still accept some pull requests.

KLAY (https://github.com/kieler/klayjs) has been succeeded by ELK, and development stopped in 2016.

ELK (https://github.com/kieler/elkjs) is currently being developed by the KIELER Project, mostly by students and professors, written in Java. They are doing a great job, but the library is written in Java and then transpiled to JavaScript. They also rely on students for help with the development, which makes progress slow.

Do you happen to know of any new projects that are actively addressing this problem?

dagrejs has a section on "Recommended Reading" which lists which algorithms from which papers they implement.

https://github.com/dagrejs/dagre/wiki#recommended-reading

Terrastruct engineer here. The visualization will be open source though I can't say for sure yet whether it'll be with our layout algorithm or with https://github.com/dagrejs/dagre.

Another option is Dagre which is apparently now abandoned.