monosi VS dagster

I also found https://github.com/monosidev/monosi but it seems there are no activities in the repository from last year.

I'm building an open source data observability platform - https://github.com/monosidev/monosi that visualizes metadata collected from data warehouses. Databricks is currently not supported (contributions welcome!), but it may help to take a look at how we approach the anomaly detection & visualization aspects.

Building an open source tool that lets you monitor PostgreSQL instances form anomalies in data coming in - https://github.com/monosidev/monosi

It’s open source, check out the repository here - https://github.com/monosidev/monosi

🎉 Congratulations, you've just set up and scheduled a data monitor on your Supabase instance. You can now add more monitors to other tables in your database. Find more information on how to use Monosi here.

Now that you’ve worked through an example using a public PostgreSQL instance, you can further extend this to your own data store. For more information, get started here.

Continuing to build out & stabilize Monosi (open source data observability) - https://github.com/monosidev/monosi

Observability: Monosi

My recommendation is to continue on with the tutorial, then look at one of the larger example projects especially the ones named “project_”, and you should understand most of it. Of what you don't understand and you're curious about, look into the relevant concept page for the functions in the docs.

I found this example that helped me - https://github.com/dagster-io/dagster/tree/master/examples/project_fully_featured/project_fully_featured

I would recommend the following: - https://www.mage.ai/ - https://dagster.io/ - https://www.prefect.io/ - https://metaflow.org/ - https://zenml.io/home

In Helm terms: there are 2 charts, namely the system: dagster/dagster (values.yaml), and the user code: dagster/dagster-user-deployments (values.yaml). Note that you have to set dagster-user-deployments.enabled: true in the dagster/dagster values-yaml to enable this.

Dagster seemed really cool when I looked into it as an alternative to airflow. I especially like the software defined assets and built-in lineage which I haven't seen in any other tool. However it seems it does not support RBAC which is a pretty big issue if you want a self-service type of architecture, see https://github.com/dagster-io/dagster/issues/2219. It does seem like it's available in their hosted version, but I wanted to run it myself on k8s.

Dagster? https://dagster.io

One that I haven't seen on here yet: dagster