Self managed Apache Airflow with Data on EKS

This page summarizes the projects mentioned and recommended in the original post on dev.to
#eks-clusters #ingress-controller #helm-charts #Kubernetes #AWS

Our great sponsors
  • InfluxDB - Access the most powerful time series database as a service
  • SonarQube - Static code analysis for 29 languages.
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • data-on-eks

    DoEKS is a tool to build, deploy and scale Data Platforms on Amazon EKS

    git clone https://github.com/awslabs/data-on-eks.git cd data-on-eks/schedulers/terraform/self-managed-airflow

  • terraform-aws-eks-blueprints

    Configure and deploy complete EKS clusters.

    Initializing modules... Downloading git::https://github.com/aws-ia/terraform-aws-eks-blueprints.git?ref=v4.15.0 for airflow_irsa... - airflow_irsa in .terraform/modules/airflow_irsa/modules/irsa Downloading registry.terraform.io/terraform-aws-modules/s3-bucket/aws 3.8.2 for airflow_s3_bucket... - airflow_s3_bucket in .terraform/modules/airflow_s3_bucket Downloading registry.terraform.io/terraform-aws-modules/rds/aws 5.6.0 for db... - db in .terraform/modules/db .. .. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.

  • InfluxDB

    Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  • airflow-dags

    Sample DAGs repo to use with Apache Airflow GitSync feature.

    You can then pop this into a browser and then login using the Airflow default user (admin/admin). You should then see the Apache Airflow 2.4.3 UI, and also have a couple of workflows (DAGs) that have been populated from the git repository configured in our code (https://github.com/Hyper-Mesh/airflow-dags)

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts