Orchestrating hybrid workflows using Amazon Managed Workflows for Apache Airflow (MWAA)

This page summarizes the projects mentioned and recommended in the original post on dev.to
Go amazon-ecs-agent amazon-ec2 docker-container amazon-linux-ami

WorkOS
Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • blogpost-airflow-hybrid

    Repo that will help you explore how to build a hybrid workflow using Apache Airflow and Amazon ECS Anywhere

    • As always, you can find the code for this walk through in this GitHub repo, blogpost-airflow-hybrid

  • amazon-ecs-agent

    Amazon Elastic Container Service Agent

    • The approach I am going to take is to create an Apache Airflow workflow (DAG) and leverage an Apache Airflow operator, ECSOperator, which allows us to launch container based images. The container based images we launch will contain our ETL code, and this will be parameterised so that we can re-use this multiple times, changing the behaviour by providing parameters during launch (for example, different SQL queries). Finally, we will use ECS Anywhere, which uses the open source amazon-ecs-agent to simplify how we can run our containers anywhere - in the Cloud, on premises or on other Clouds.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts