docker-airflow

Docker Apache Airflow (by puckel)

Docker-airflow Alternatives

Similar projects and alternatives to docker-airflow

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better docker-airflow alternative or higher similarity.

docker-airflow reviews and mentions

Posts with mentions or reviews of docker-airflow. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2022-02-20.
  • Kubernetes deployment read-only filesystem error
    1 project | /r/codehunter | 5 Sep 2022
    I am facing an error while deploying Airflow on Kubernetes (precisely this version of Airflow https://github.com/puckel/docker-airflow/blob/1.8.1/Dockerfile) regarding writing permissions onto the filesystem.
  • How to use virtual environment in airflow DAGS?
    1 project | /r/apache_airflow | 23 May 2022
    I used https://github.com/puckel/docker-airflow to setup the airflow and I moved my python scripts inside the dags directory but now they won't execute because I can't access the installed libraries in the virtual environment. How can i find a workaround for this?
  • Amount of effort to stand up, integrate and manage a small airflow implementation
    2 projects | /r/dataengineering | 20 Feb 2022
    Used a custom version of Puckel Airflow Docker image (Spent a lot of time customising to our needs, but default Airflow container should still work)
  • The Unbundling of Airflow
    3 projects | news.ycombinator.com | 15 Feb 2022
    I understand it is subjective. But I use a forked version of https://github.com/puckel/docker-airflow on our managed K8s cluster and it points to a cloud managed Postgres. It has worked pretty well for over 3 years with no-one actually managing it from an infra POV. YMMV. This is driving a product whose ARR is well in the 100s of Millions.

    If you have simple needs that are more or less set, I agree Airflow is overkill and a simple Jenkins instance is all you need.

  • Airflow v1 to v2 - Recommendations / RoX
    1 project | /r/dataengineering | 9 Feb 2022
    So were running Airflow v1 (based on this docker compose) with a sequential executor running on an on prem OpenShift v3 setup. We have a new / free resource coming and have planned to use them to reinitiate a complete new version utilizing OpenShift v4 (also on prem but not managed by us) and upgrade in parallel to Airflow v2. The question is if anyone has any strong recommendations on a good docker compose file they would look at and any views on celery / kubernets workers. We're not a huge team but have a bit of experience up our sleeves now so was more after some guidance or thoughts if others have gone down similar paths. Thanks!
  • Can someone help me understand the difference between the the docker-compose files?
    1 project | /r/dataengineering | 9 Sep 2021
    version: '3' services: postgres: image: postgres:9.6 environment: - POSTGRES_USER=airflow - POSTGRES_PASSWORD=airflow - POSTGRES_DB=airflow ports: - "5432:5432" webserver: image: puckel/docker-airflow:1.10.1 build: context: https://github.com/puckel/docker-airflow.git#1.10.1 dockerfile: Dockerfile args: AIRFLOW_DEPS: gcp_api,s3 PYTHON_DEPS: sqlalchemy==1.2.0 restart: always depends_on: - postgres environment: - LOAD_EX=n - EXECUTOR=Local - FERNET_KEY=jsDPRErfv8Z_eVTnGfF8ywd19j4pyqE3NpdUBA_oRTo= volumes: - ./examples/intro-example/dags:/usr/local/airflow/dags # Uncomment to include custom plugins # - ./plugins:/usr/local/airflow/plugins ports: - "8080:8080" command: webserver healthcheck: test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"] interval: 30s timeout: 30s retries: 3
  • How should I get started with CI/CD ? (new to data engineering)
    1 project | /r/dataengineering | 10 Apr 2021
    As for learning, learn how to build and use docker containers. For airflow, take a look a https://github.com/puckel/docker-airflow and see how to add you pipelines to that container. Then learn how to do CI/CD for docker containers (tons of tutorials). Then learn to deploy containers, you can use aws ecs.
  • Interview - take home project on data ingestion, warehouse design, basic analytics and conceptual using python and sql.
    1 project | /r/dataengineering | 25 Mar 2021
    Usually googling the software you want + docker will get you what you need. For that particular project, I used https://github.com/puckel/docker-airflow to help set up a local airflow instance.
  • ETL com Apache Airflow, Web Scraping, AWS S3, Apache Spark e Redshift | Parte 1
    3 projects | dev.to | 4 Jan 2021
    A imagem do docker utilizada foi a puckel/docker-airflow onde acrescentei o BeautifulSoup como dependência para criação da imagem em minha máquina.
  • How we evolved our data engineering workflow day by day
    1 project | dev.to | 7 Dec 2020
    We used to schedule and monitor workflows tool airflow as our ELT processor and have to extract data from SQL and No-SQL databases to load them into the warehouse. Our airflow deployment was done through docker, for more details checkout puckel/airflow. Currently, we are adopting our image to the official docker images.
  • A note from our sponsor - InfluxDB
    www.influxdata.com | 27 May 2024
    Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Stats

Basic docker-airflow repo stats
10
3,733
0.0
about 1 year ago

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com