Working with Managed Workflows for Apache Airflow (MWAA) and Amazon Redshift

• cdk-mwaa-redshift

  1 0 3.5 Python

  A repo showing how to quickly get MWAA and Redshift up and running with a few sample DAGs

git clone https://github.com/094459/cdk-mwaa-redshift

• blog-mwaa-redshift

  2 1 4.1 Python

With that out of the way, how would you use the PythonOperator to work with Redshift? You would create a function within your DAG that has the logic for what you are looking to do, and then invoke via the operator. In this example DAG which I put together a few years back before I had discovered the Redshift operators, you can see how I created tables in Redshift.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Airflow

  169 34,485 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

You can actually setup and delete new Redshift clusters using Apache Airflow. We can see in the example_dags of a DAG that does a complete setup and delete of a Redshift cluster. There are a few things to think about however.

Suggest a related project