awesome-data-catalogs VS grai-core

Check out this GitHub awesome list of Data Catalogs.

Maybe what you want i some specification from which you can build something? In that way, perhaps this can help you https://github.com/opendatadiscovery/awesome-data-catalogs. Airflow uses OpenLineage as a way to send their metadata, and Marquez collects them to show them in their UI (https://openlineage.io/docs/guides/airflow), so I suppose you would want to do something similar? But maybe in that GitHub you can find other specifications that can help you better.

In order to gain a understanding of the tables and their contents in our company, I have implemented one of the existing [data discovery platforms](https://github.com/opendatadiscovery/awesome-data-catalogs) (in my case [Amundsen](https://www.amundsen.io/))). Unfortunately, Amundsen can only display the tables it has access to.

I got nice data catalog summary in case anyone would be interested - https://github.com/opendatadiscovery/awesome-data-catalogs. It is probably biased since author is also author of one of the data catalogs, but still can be quite useful :)

Hi, I would like to know is there any opensource data catalog systems that targets machine learning applications (datasets (unstructral, e.g., text, image, and video) and models) as the citizen?

I have read the awesome-data-catalogs ([1]) list but found none of them is treating ML as 1st cizten and the support for datasets and models are not specific enough.

[1]: https://github.com/opendatadiscovery/awesome-data-catalogs

Elastic v2 if one is interested in such things: https://github.com/grai-io/grai-core/blob/v0.1.33/LICENSE

I’m not sure if this is precisely what you’re looking for but Grai might serve your needs. The backend data model allows you to push any arbitrary metadata you want / need onto the lineage graph and retrieve it either through the rest or graph API. I’m one of the authors so happy to answer any questions you might have.

We've been looking at building something like this for Grai specifically to support Airflow but haven't yet prioritized it.

We were frustrated because although we had tests in our data warehouse, they only notified us after an outage occurred. What we needed was a way to detect changes during CI/CD, so we could fix things before they impacted production. So we developed Grai, as an open-source data lineage toolkit pre-built integrations for the most common data stores and designed to work with CI tools, like Github Actions.