Our great sponsors
-
mara-pipelines
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
-
etl-markup-toolkit
ETL Markup Toolkit is a spark-native tool for expressing ETL transformations as configuration
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
The closest I've found is Mara but not what I'm after.
Not sure if it meets your exact requirements, but I maintain an open source project that enables spark transformations as configuration, and part of that capability is reporting, including logging of columns in vs columns out, row counts, etc... It's very early stage but perhaps could be useful - https://github.com/leozqin/etl-markup-toolkit
Related posts
- Data sources episode 2: AWS S3 to Postgres Data Sync using Singer
- How do you serialize and save "transformations" in your pipeline?
- Alternative tools to DBT / SQL and Python for writing business logic? Trying to prevent creating a mountain of undocumented spaghetti
- Getting started with Azure Data Explorer and Azure Synapse Analytics for Big Data processing
- ETL Markup Toolkit - a spark native tool for describing etl transformations as configuration