finch VS aws-mwaa-local-runner

I was getting hundreds of permissions errors. Oh no, I knew this was going too well. Looking at the current issues within the Finch GitHub repo, I found an issue that I thought would help resolve this problem. Looking at this I created a new docker-compose file to take into consideration some of the comments, as well as adding a new step.

You can also try finch from AWS which combines a similar stack

https://github.com/runfinch/finch

I work at AWS and helped with testing finch during development

I've not used this myself, but they are pushing this at work instead of having to pay for Docker Desktop - Check out Finch - OpenSource

Lima is still quite new and I am even newer to it so bear in mind that there may be better ways to wield this weapon. There is also a lot of development in this space with the likes of Finch, Multipass and others offering potential alternatives or simplifications to this approach so watch this space!

As some of you may know, I have been creating content on Apache Airflow for a few years now. One of the open source projects that AWS has produced to make it easier for developers to get started with Apache Airflow, is mwaa-local-runner. If you have seen me at an event, it is likely you will have seen my live coding/demos, where I use this project. It is awesome!

As of writing, there is a small "feature" with mwaa-local-runner, which means we have to make s small change to our mwaa-local-runner before we can use this. We will need to modify the "entrypoint.sh" script in the docker/scripts folder, and update it by adding a new line " source stored_env" between the execute_start_script and the ariflow db init, as follows:

git clone https://github.com/aws/aws-mwaa-local-runner.git

I’d also take a look into the AWS data wrangler python package and the local airflow runner https://github.com/aws/aws-mwaa-local-runner. This builds a local airflow server to play with. You can start using Airflow, learn about dags and tasks, passing data through XCOMs, etc.

I am running aws-mwaa-local-runner in order to run a local Apache Airflow environment (in Docker for Windows).

I wonder if you encountered this issue when trying to build image from aws-mwaa-local-runner repository. I'm using their release v2.0.2. Apparently it has problem with Flask-OpenID 1.2.5. (See output below.)

Here’s a super easy way to replicate a production airflow environment: https://github.com/aws/aws-mwaa-local-runner. Basically this is the recommended local dev environment when using AWS managed airflow service (MWAA).