data-drift VS lakeFS

Think of Datadrift as a simple & open-source Monte Carlo for the semantic layer era. The repo is at https://github.com/data-drift/data-drift

Datadrift started as an internal tool built at our former company, a large European B2B Fintech. We had data reliability challenges impacting key metrics used for financial and regulatory reporting.

However, when we tried existing data quality tools we where always frustrated. They provide row-level static testing (eg. uniqueness or nullness) which does not address time-varying metrics like revenues. And commercial observability solutions costs $manyK a month and brings compliance and security overhead.

We designed Datadrift to solve these problems. Datadrift works by simply adding a monitor where your metric is computed. It then understands how your metric is computed and on which upstream tables it depends. When an issue occurs, it pinpoints exactly which rows have been updated and introducing the change.

You can also set up alerting and customise it. For example, you can decide to open and assign an Github issue to the analyst owning the revenue metric when a +10% change is detected. We tried to make it easy to customise and developer friendly.

We are thinking of adding features around root cause analysis automation/issues pattern analysis to help data teams improve metrics quality overtime. We’d love to hear your feature requests.

Datadrift is built with Python and Go, and licensed under GPL. Our docs are here: https://github.com/data-drift/data-drift?tab=readme-ov-file#...

Dev set up and demo : https://app.claap.io/sammyt/drift-db-demo-a18-c-ApwBh9kt4p-0...

We’re very eager to get your feedback!

I have already started working on it, I started in Go for some part, but I needed python to deploy a Pypi lib. Now its hybrid, and I prefer working with go 😬 but the most rational thinking leads to python.

We used static testing framework like great expectations but that was not enough. We did not have the budget for the big data observability players like Monte Carlo, so we kept it simple.

Repo if interested: https://github.com/data-drift/data-drift

(Disclaimer: I am focusing full time on this project to see if it's an interesting business opportunity. It's 100% open-source -- feedback welcome!)

https://app.data-drift.io/42527392/Lucasdvrs/dbt-datagit/ove...

You can "technically" install it by yourself, but tbh our focus are on the features, not the adoption. If you are interested it takes roughly 1 hour to configure (choose the data you want to observe, run a python function, install a Github app, add a configuration file), contact us.

The repo: https://github.com/data-drift/data-drift

Roast me

# Download the LakeFS binary wget https://github.com/treeverse/lakeFS/releases/latest/download/lakefs # Make the binary executable chmod +x lakefs # Initialize LakeFS with S3 as the storage backend ./lakefs init --backend s3 --s3-gateway-endpoint --s3-region --s3-force-path-style --s3-access-key --s3-secret-key

Might want to look at purpose built tools for that such as lakeFS (https://github.com/treeverse/lakeFS/)

* Disclaimer: I'm one of the creators/maintainers of the project.

Might want to checkout lakeFS: https://github.com/treeverse/lakeFS

(full disclosure: I'm one of the creators)

Take a look at https://github.com/treeverse/lakeFS -

I mean if you're ready to adopt a new framework into your ecosystem this is one of the major usecases for LakeFS.