DataProfiler VS elementary

It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler

For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.

Great call out -- would you be willing to write up an issue for that on the repo? Thank you! https://github.com/capitalone/DataProfiler/issues/new/choose

Check out https://github.com/elementary-data/elementary

In this case, do https://github.com/elementary-data/elementary or https://greatexpectations.io help?

That makes explain little easier to read. No graph though. Also https://github.com/elementary-data/elementary should know howto draw pretty graphs for data lineage ( ie. what columns comes where and is used how)

there is elementary lineage / elementary-data which seems to be good try to solve those problem, i havent tested it well https://github.com/elementary-data/elementary

Some things you might be interested in are re_data and Elementary Data.

Hi all, As part of building Elementary (open source data reliability), we implemented support of a new Snowflake feature (write operations in the access_history view). The change they made is most useful for understanding data lineage, which we solve (among other use cases :)).

I recently stumbled on an open-source tool with a similar premise: https://github.com/elementary-data/elementary-lineage

you can check it out

Hi! we are working on an open source data lineage solution that might be helpful for your use case to learn the relationship between tables, we don't support column level just yet but we are working on it. Please let me know if we can help somehow and feel free to check it out here - https://github.com/elementary-data/elementary-lineage