data-lineage
grai-core
| data-lineage | grai-core | |
|---|---|---|
| 1 | 6 | |
| 297 | 270 | |
| 1.3% | 1.5% | |
| 2.0 | 9.5 | |
| 10 months ago | 6 days ago | |
| Python | Python | |
| MIT License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
data-lineage
-
Column level lineage
Link: https://github.com/tokern/data-lineage
grai-core
-
Launch HN: Grai (YC S22) – Open-Source Data Observability Platform
Elastic v2 if one is interested in such things: https://github.com/grai-io/grai-core/blob/v0.1.33/LICENSE
-
Standalone lineage tool
I’m not sure if this is precisely what you’re looking for but Grai might serve your needs. The backend data model allows you to push any arbitrary metadata you want / need onto the lineage graph and retrieve it either through the rest or graph API. I’m one of the authors so happy to answer any questions you might have.
-
Data Load Diagram
We've been looking at building something like this for Grai specifically to support Airflow but haven't yet prioritized it.
-
Grai, a self-hosted data lineage tool. Test downstream impact of data migration changes
We were frustrated because although we had tests in our data warehouse, they only notified us after an outage occurred. What we needed was a way to detect changes during CI/CD, so we could fix things before they impacted production. So we developed Grai, as an open-source data lineage toolkit pre-built integrations for the most common data stores and designed to work with CI tools, like Github Actions.
What are some alternatives?
lux - Automatically visualize your pandas dataframe via a single print! 📊 💡
dbt-snowflake-monitoring - A dbt package from SELECT to help you monitor Snowflake performance and costs
lux - 👾 Fast and simple video download library and CLI tool written in Go
awesome-data-catalogs - 📙 Awesome Data Catalogs and Observability Platforms.
sqllineage - SQL Lineage Analysis Tool powered by Python
jupysql - Better SQL in Jupyter. 📊
ipython - Official repository for IPython itself. Other repos in the IPython organization contain things like the website, documentation builds, etc.
MindsDB - The platform for customizing AI from enterprise data
django-pgschemas - Django multi-tenancy through Postgres schemas
sqlparse - A non-validating SQL parser module for Python
ibis - the portable Python dataframe library
django-pg-upsert - Support Postgres native upsert (INSERT ... ON CONFLICT) for django