Our great sponsors
-
aws-mwaa-local-runner
This repository provides a command line interface (CLI) utility that replicates an Amazon Managed Workflows for Apache Airflow (MWAA) environment locally.
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
finch-local-runner
A repo that provides a way of running mwaa-local-runner to use Finch rather than Docker and Docker Compose.
As some of you may know, I have been creating content on Apache Airflow for a few years now. One of the open source projects that AWS has produced to make it easier for developers to get started with Apache Airflow, is mwaa-local-runner. If you have seen me at an event, it is likely you will have seen my live coding/demos, where I use this project. It is awesome!
I was getting hundreds of permissions errors. Oh no, I knew this was going too well. Looking at the current issues within the Finch GitHub repo, I found an issue that I thought would help resolve this problem. Looking at this I created a new docker-compose file to take into consideration some of the comments, as well as adding a new step.
I hope this post was useful, and that for those of you who are looking to use open source tools like Finch to manage your container development processes, you will get some ideas of how easy it can be. I have created a GitHub repo that shares the configuration files I used to get mwaa-local-runner to work with Finch.