-
blogpost-airflow-hybrid
Repo that will help you explore how to build a hybrid workflow using Apache Airflow and Amazon ECS Anywhere
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows (by 094459)
My solution was to use Apache Airflow and create a new workflow to orchestrate this. I planned to create an ETL script, and ensure the script can take parameters, to maximise reuse and flexibility.
With permission granted, the first step is to head over to the Apache Airflow project and look for information on how to contribute. It is a good practice to create a CONTRIBUTING document in your project to help guide new contributors as well as ensure consistency for those who might already be familiar.
Forking Apache Airflow is simple enough, as GitHub makes this as simple as a couple of clicks. I now have my own Apache Airflow fork for for this work. The only thing I needed to make sure was that it was the right branch (latest, main) and not any of the other branches (I have made that mistake before!)
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
Related posts
-
Building in Public: Leveraging Tublian's AI Copilot for My Open Source Contributions
-
Navigating Week Two: Insights and Experiences from My Tublian Internship Journey
-
Airflow VS quix-streams - a user suggested alternative
2 projects | 7 Dec 2023 -
Best ETL Tools And Why To Choose
-
Simplifying Data Transformation in Redshift: An Approach with DBT and Airflow