Top 23 Python Airflow Projects

Airflow

• Add a project

• Airflow

  171 35,036 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

Project mention: 10 Open Source Tools for Building MLOps Pipelines | dev.to | 2024-06-06

An integral part of an ML project is data acquisition and data transformation into the required format. This involves creating ETL (extract, transform, load) pipelines and running them periodically. Airflow is an open source platform that helps engineers create and manage complex data pipelines. Furthermore, the support for Python programming language makes it easy for ML teams to adopt Airflow.

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

• elyra

  2 1,791 7.2 Python

  Elyra extends JupyterLab with an AI centric approach.

  

• airflow-maintenance-dags

  1 1,616 0.0 Python

  A series of DAGs/Workflows to help maintain the operation of Airflow

  

• Udacity-Data-Engineering-Projects

  5 1,363 0.0 Python

  Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Project mention: Pitanje za data engineering? | /r/programiranje | 2023-06-30

• DataEngineeringProject

  5 1,032 0.0 Python

  Example end to end data engineering project.

• energy-forecasting

  1 804 8.5 Python

  🌀 𝗧𝗵𝗲 𝗙𝘂𝗹𝗹 𝗦𝘁𝗮𝗰𝗸 𝟳-𝗦𝘁𝗲𝗽𝘀 𝗠𝗟𝗢𝗽𝘀 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 | 𝗟𝗲𝗮𝗿𝗻 𝗠𝗟𝗘 & 𝗠𝗟𝗢𝗽𝘀 for free by designing, building and deploying an end-to-end ML batch system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 2.5 𝘩𝘰𝘶𝘳𝘴 𝘰𝘧 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 & 𝘷𝘪𝘥𝘦𝘰 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴

Project mention: Full Stack 7-Steps MLOps Framework – Learn ML and MLOps by Building an ML System | news.ycombinator.com | 2023-09-26

• astronomer-cosmos

  8 490 9.5 Python

  Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code

  

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• streamify

  4 474 0.0 Python

  A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

• airflow-dbt

  1 383 0.0 Python

  Apache Airflow integration for dbt

  

• astro-sdk

  7 327 8.5 Python

  Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

  

• airflow-chart

  1 267 7.7 Python

  A Helm chart to install Apache Airflow on Kubernetes

  

• airflow-docker

  2 237 3.0 Python

  Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j) (by coder2j)

• airflowctl

  1 173 7.7 Python

  A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects

Project mention: A look at airflowctl, a tool to help developers manage Apache Airflow projects | dev.to | 2023-08-14

NOTE! I found a small issue in that when you run in background mode, it creates a file (.airflowctl/.background_process_ids) which contains the parent PID. The PID was always off, so I needed to manually edit this. I have created an issue here so if this happens to you, follow that.

• audiophile-e2e-pipeline

  3 170 0.0 Python

  Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard.

• ask-astro

  2 164 8.8 Python

  An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer

  

Project mention: Show HN: Open-source implementation of A16Z’s LLM architecture | news.ycombinator.com | 2023-09-22

• airflow-testing-ci-workflow

  2 84 0.0 Python

  (project & tutorial) dag pipeline tests + ci/cd setup

• magniv-core

  1 75 0.0 Python

  Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.

  

• airflow-elt-blueprint

  1 47 2.9 Python

  A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.

• soopervisor

  3 43 5.0 Python

  ☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.

  

• pyjaws

  4 39 6.4 Python

  PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows

Project mention: PyJaws: A Pythonic Way to Define Databricks JaWs (Jobs and Workflows) | /r/pythontips | 2023-09-25

🐍 Install 𝐩𝐲𝐣𝐚𝐰𝐬 from PyPi: https://pypi.org/project/pyjaws/

• dbt-cloud-plugin

  2 38 0.0 Python

  DBT Cloud Plugin for Airflow

• tfl-bikes-data-pipeline

  1 37 0.0 Python

  Processing TFL data for bike usage with Google Cloud Platform.

• typhoon-orchestrator

  14 29 0.0 Python

  Create elegant data pipelines and deploy to AWS Lambda or Airflow

  

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Python Airflow related posts

• Building in Public: Leveraging Tublian's AI Copilot for My Open Source Contributions

  1 project | dev.to | 12 Feb 2024

• Navigating Week Two: Insights and Experiences from My Tublian Internship Journey

  1 project | dev.to | 31 Dec 2023

• Airflow VS quix-streams - a user suggested alternative

  2 projects | 7 Dec 2023

• Best ETL Tools And Why To Choose

  1 project | /r/tactionsoftware | 11 Nov 2023

• Simplifying Data Transformation in Redshift: An Approach with DBT and Airflow

  2 projects | dev.to | 7 Nov 2023

• Share Your favorite python related software!

  1 project | /r/Python | 1 Oct 2023

• Show HN: Open-source implementation of A16Z’s LLM architecture

  1 project | news.ycombinator.com | 22 Sep 2023
• A note from our sponsor - Scout Monitoring
  www.scoutapm.com | 16 Jun 2024

  Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today. Learn more →

Index

Sponsored

Free Django app performance insights with Scout Monitoring

Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

www.scoutapm.com

Do not miss the trending Python projects with our weekly report!