amundsen VS datahub

We'll be using WSL2 for this guide, and we'll start by cloning this repo and its submodules:

You can use Amundsen data builder to send data to Apache Atlas, https://github.com/amundsen-io/amundsen/blob/main/databuilder/example/scripts/sample_atlas_search_extractor.py If you don’t have to configure Apache Atlas then why not, but the server side validation the last time when I used it was absent. You couldn’t validate the JSON body sent to the REST API endpoints.

Other than that, maybe you could try amundsen (https://github.com/amundsen-io/amundsen/issues/608) which now has a connector to extract delta lake metadata via Spark.

How does it differ from something like Amundsen : https://github.com/amundsen-io/amundsen

Metadata Engine: - Datahub https://github.com/linkedin/datahub - Amundsen https://github.com/amundsen-io/amundsen/ - Marquez https://marquezproject.github.io/ - Egeria - Open Metadata and Governance https://egeria.odpi.org

A final category worth highlighting is Discovery, where it seems every notable company developed an internal Data Catalogue tool that now is available as an open-source or paid service. Some examples are Amundsen (Lyft), Datahub (LinkedIn), Metacat (Netflix), Databook (Uber), and Dataportal (Airbnb).

Sounds like you are looking for a data catalog tool instead of db schema management tool. You can check out Amundsen (https://www.amundsen.io/), DataHub (https://datahubproject.io/)

If you are looking for schema change management tool, then you can check out Bytebase (bytebase.com). But it can't answer questions like "which collections contain links to bigmongo.user.id?"

IIUC DataHub (open source project out of LinkedIn) might be relevant here

I’m a huge fan of DataHub, the open source data catalogue spun out of LinkedIn, but it’s best thought of as an observability layer for data assets that can be shared by data engineers and analyst-types. For data users: it’s a stellar search/discovery interface (what datasets are there on this keyword, which are most broadly used across the organization, what downstream products are made with this data, what’s it usually joined to, are it’s upstream pipelines reliable). For data engineers, it’s a comprehensive asset cataloger, crawling your warehouse, orchestrator, modeling layers, features, and reports, matching the lineage into a graph where it can.

If you are looking for a data cataloguing solution, look at Datahub. Haven't used it, but heard good things about it.

What I am looking for is a solution (similar to Amundsen or [Datahub](https://datahubproject.io/)) that also allows to add tables and their metadata manually.

Hello everyone, I'm looking for an open-source data lineage app (e.g. tokern, datahubproject, openmetadata).

What about DataHub? Haven't really used it but I'm actively reading about it and about to use it for some light documentation for some small pipelines.