Our great sponsors
-
airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
At the same time, we have been working with some members of the community on a new framework for building taps and targets: https://gitlab.com/meltano/meltano/-/issues/2401, which we have decided to call the Singer SDK: https://gitlab.com/meltano/singer-sdk. We are moving as many Singer specification-specific details around things like incremental state replication and stream/field selection into the framework, so that individual taps only need to worry about getting the data from the source and can be expected to behave more consistently and correctly across the board.
Note: I am one of Airbyte's maintainers.
I actually built a POC of that earlier this year that uses https://gitlab.com/meltano/gitlab-ci-scheduler: https://gitlab.com/meltano/meltano/-/merge_requests/1620, demo: https://gitlab.com/DouweM/orchestrator-gitlab-test/pipeline_schedules
This is something I've been looking at as well. I've recently found a python-based implementation here driven by Github user `wintersrd` which might be a good candidate for adoption by meltano as that tap reaches maturity.