Our great sponsors
-
airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
I've stress tested Airbyte a number of times, many of their connectors don't work, there are issues galore with some connectors only supporting full refreshes which screws up many API calls for obvious reasons around rate limits, etc, and if you don't believe me about how unstable of a product this is, I encourage you to do look at Airbyte's public issue log on GH. It is comedy. https://github.com/airbytehq/airbyte/issues
You can get pretty good results with Ploomber (https://github.com/ploomber/ploomber), some out of the box examples is our ml use cases the ability to build modular pipelines. In addition you can deploy into multiple platforms. The learning curve is very short since you get to keep your existing processes like using Jupyter/IDE and not moving into a designated UI.