docker-flask-example VS Airflow

I've been maintaining my Build a SAAS App with Flask video course[0] for 8 years. It has gone from pre-1.0 to 2.3 and has been recorded twice with tons of incremental updates added over the years to keep things current.

In my opinion tutorial creators should pin their versions so that anyone taking the course or going through the tutorial will have a working version that matches the video or written material.

I'm all for keeping things up to date and do update things every few months but rolling updates don't tend to work well for tutorials because sometimes a minor version requires a code change or covering new concepts. As a tutorial consumer it's frustrating when the content doesn't match the source code unless it's nothing but a version bump.

I've held off upgrading Flask to 3.0 and Python 3.12 due to these open issues with 3rd party dependencies https://github.com/nickjj/docker-flask-example/issues/17.

[0]: https://buildasaasappwithflask.com/

I usually end up with project specific "run" scripts which are just shell scripts so I can do things like `./run shell` to drop into the shell of a container, or `./run rails db:migrate` to run a command in a container.

Here's a few project specific examples. They all have similar run scripts:

    - https://github.com/nickjj/docker-flask-example

There's a lot of "tool" selections in that repo.

If anyone is looking for ready to go web app examples aimed at both development and production, I maintain:

    - https://github.com/nickjj/docker-flask-example

Personally I maintain https://github.com/nickjj/docker-flask-example. There's also https://github.com/nickjj/build-a-saas-app-with-flask if you want more opinions.

This is what I do except I use a shell script instead of a Makefile.

A working example of this is at: https://github.com/nickjj/docker-flask-example/blob/912388f3...

Those ./run ci:XXX commands are in: https://github.com/nickjj/docker-flask-example/blob/912388f3...

I like it because if CI ever happens to be down I can still run that shell script locally.

> I've encountered a code written in the 12factor style of using environment variables for configuration, and in that particular case there was no validation nor documentation of the configuration options. Is this typical?

I don't know about typical, it comes down to how your team values the code they write.

You can have a .env.example file commit to version control which explains every option in as much or as little detail as you'd like. For my own personal projects, I tend to document this file like this https://github.com/nickjj/docker-flask-example/blob/main/.en....

I did this for a while but make isn't well suited for this use case. What I end up doing is have a shell script with a bunch of functions in it. Functions automatically becomes a callable a command (with a way to make private functions if you want) with pretty much no boiler plate.

The benefit of this is it's just shell scripting so you can use shell features like $@ to pass args to another command or easily source and deal with env vars.

I've written about this process at https://nickjanetakis.com/blog/replacing-make-with-a-shell-s... and an example file is here https://github.com/nickjj/docker-flask-example/blob/main/run.

There's: https://github.com/nickjj/docker-flask-example

Contributing to Apache Airflow's open-source project immersed me in collaborative coding. Experienced maintainers rigorously reviewed my contributions, providing constructive feedback. This ongoing dialogue refined the codebase and honed my understanding of best practices.

In week Two, I contributed to the Apache Airflow repository.

Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. The platform features a web-based user interface and a command-line interface for managing and triggering workflows.

Airflow is the most widely used and well-known tool for orchestrating data workflows. It allows for efficient pipeline construction, scheduling, and monitoring.

AIRFLOW This is more of a library in my opinion, but Airflow has become an essential tool for scheduling in my work. All our ML training pipelines are ordered and scheduled with Airflow and it works seamlessly. The dashboard provided is also fantastic!

I agree there are many options in this space. Two others to consider:

- https://airflow.apache.org/

- https://github.com/spotify/luigi

There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file showing up in a directory…

You can try and contribute some new connectors/operators for workflow managers like Airflow or Airbyte