Docker-airflow Alternatives

Reviews and mentions

• Can someone help me understand the difference between the the docker-compose files?
  reddit.com/r/dataengineering | 2021-09-09

  version: '3' services: postgres: image: postgres:9.6 environment: - POSTGRES_USER=airflow - POSTGRES_PASSWORD=airflow - POSTGRES_DB=airflow ports: - "5432:5432" webserver: image: puckel/docker-airflow:1.10.1 build: context: https://github.com/puckel/docker-airflow.git#1.10.1 dockerfile: Dockerfile args: AIRFLOW_DEPS: gcp_api,s3 PYTHON_DEPS: sqlalchemy==1.2.0 restart: always depends_on: - postgres environment: - LOAD_EX=n - EXECUTOR=Local - FERNET_KEY=jsDPRErfv8Z_eVTnGfF8ywd19j4pyqE3NpdUBA_oRTo= volumes: - ./examples/intro-example/dags:/usr/local/airflow/dags # Uncomment to include custom plugins # - ./plugins:/usr/local/airflow/plugins ports: - "8080:8080" command: webserver healthcheck: test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"] interval: 30s timeout: 30s retries: 3

• How should I get started with CI/CD ? (new to data engineering)
  reddit.com/r/dataengineering | 2021-04-10

  As for learning, learn how to build and use docker containers. For airflow, take a look a https://github.com/puckel/docker-airflow and see how to add you pipelines to that container. Then learn how to do CI/CD for docker containers (tons of tutorials). Then learn to deploy containers, you can use aws ecs.

• Interview - take home project on data ingestion, warehouse design, basic analytics and conceptual using python and sql.
  reddit.com/r/dataengineering | 2021-03-25

  Usually googling the software you want + docker will get you what you need. For that particular project, I used https://github.com/puckel/docker-airflow to help set up a local airflow instance.

• ETL com Apache Airflow, Web Scraping, AWS S3, Apache Spark e Redshift | Parte 1
  dev.to | 2021-01-04

  A imagem do docker utilizada foi a puckel/docker-airflow onde acrescentei o BeautifulSoup como dependência para criação da imagem em minha máquina.

• How we evolved our data engineering workflow day by day
  dev.to | 2020-12-07

  We used to schedule and monitor workflows tool airflow as our ELT processor and have to extract data from SQL and No-SQL databases to load them into the warehouse. Our airflow deployment was done through docker, for more details checkout puckel/airflow. Currently, we are adopting our image to the official docker images.