wordpress-docker-compose VS docker-airflow

If it is so 'easy' please weigh in on this thread on github on how to do it.... because that is the only thing close to a solution and it doesn't work.... https://github.com/nezhar/wordpress-docker-compose/issues/68 if you have all the answers to show us how easy this, then please enlighten us!

How to Install Wordpress with Docker Compose

I've been using: https://github.com/nezhar/wordpress-docker-compose with docker-compose and it works ok, but there are a few quirks especially with NixOS I can't seem to figure out.

I am facing an error while deploying Airflow on Kubernetes (precisely this version of Airflow https://github.com/puckel/docker-airflow/blob/1.8.1/Dockerfile) regarding writing permissions onto the filesystem.

I used https://github.com/puckel/docker-airflow to setup the airflow and I moved my python scripts inside the dags directory but now they won't execute because I can't access the installed libraries in the virtual environment. How can i find a workaround for this?

Used a custom version of Puckel Airflow Docker image (Spent a lot of time customising to our needs, but default Airflow container should still work)

I understand it is subjective. But I use a forked version of https://github.com/puckel/docker-airflow on our managed K8s cluster and it points to a cloud managed Postgres. It has worked pretty well for over 3 years with no-one actually managing it from an infra POV. YMMV. This is driving a product whose ARR is well in the 100s of Millions.

If you have simple needs that are more or less set, I agree Airflow is overkill and a simple Jenkins instance is all you need.

So were running Airflow v1 (based on this docker compose) with a sequential executor running on an on prem OpenShift v3 setup. We have a new / free resource coming and have planned to use them to reinitiate a complete new version utilizing OpenShift v4 (also on prem but not managed by us) and upgrade in parallel to Airflow v2. The question is if anyone has any strong recommendations on a good docker compose file they would look at and any views on celery / kubernets workers. We're not a huge team but have a bit of experience up our sleeves now so was more after some guidance or thoughts if others have gone down similar paths. Thanks!

version: '3' services: postgres: image: postgres:9.6 environment: - POSTGRES_USER=airflow - POSTGRES_PASSWORD=airflow - POSTGRES_DB=airflow ports: - "5432:5432" webserver: image: puckel/docker-airflow:1.10.1 build: context: https://github.com/puckel/docker-airflow.git#1.10.1 dockerfile: Dockerfile args: AIRFLOW_DEPS: gcp_api,s3 PYTHON_DEPS: sqlalchemy==1.2.0 restart: always depends_on: - postgres environment: - LOAD_EX=n - EXECUTOR=Local - FERNET_KEY=jsDPRErfv8Z_eVTnGfF8ywd19j4pyqE3NpdUBA_oRTo= volumes: - ./examples/intro-example/dags:/usr/local/airflow/dags # Uncomment to include custom plugins # - ./plugins:/usr/local/airflow/plugins ports: - "8080:8080" command: webserver healthcheck: test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"] interval: 30s timeout: 30s retries: 3

As for learning, learn how to build and use docker containers. For airflow, take a look a https://github.com/puckel/docker-airflow and see how to add you pipelines to that container. Then learn how to do CI/CD for docker containers (tons of tutorials). Then learn to deploy containers, you can use aws ecs.

Usually googling the software you want + docker will get you what you need. For that particular project, I used https://github.com/puckel/docker-airflow to help set up a local airflow instance.

A imagem do docker utilizada foi a puckel/docker-airflow onde acrescentei o BeautifulSoup como dependência para criação da imagem em minha máquina.

We used to schedule and monitor workflows tool airflow as our ELT processor and have to extract data from SQL and No-SQL databases to load them into the warehouse. Our airflow deployment was done through docker, for more details checkout puckel/airflow. Currently, we are adopting our image to the official docker images.