Top 23 Python Airflow Projects

• Airflow

  169 34,570 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

Project mention: Building in Public: Leveraging Tublian's AI Copilot for My Open Source Contributions | dev.to | 2024-02-12

Contributing to Apache Airflow's open-source project immersed me in collaborative coding. Experienced maintainers rigorously reviewed my contributions, providing constructive feedback. This ongoing dialogue refined the codebase and honed my understanding of best practices.

• elyra

  2 1,776 6.3 Python

  Elyra extends JupyterLab with an AI centric approach.

  

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• airflow-maintenance-dags

  1 1,601 0.0 Python

  A series of DAGs/Workflows to help maintain the operation of Airflow

  

• Udacity-Data-Engineering-Projects

  5 1,295 0.0 Python

  Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Project mention: Pitanje za data engineering? | /r/programiranje | 2023-06-30

• DataEngineeringProject

  5 985 0.0 Python

  Example end to end data engineering project.

• energy-forecasting

  1 789 8.5 Python

  🌀 𝗧𝗵𝗲 𝗙𝘂𝗹𝗹 𝗦𝘁𝗮𝗰𝗸 𝟳-𝗦𝘁𝗲𝗽𝘀 𝗠𝗟𝗢𝗽𝘀 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 | 𝗟𝗲𝗮𝗿𝗻 𝗠𝗟𝗘 & 𝗠𝗟𝗢𝗽𝘀 for free by designing, building and deploying an end-to-end ML batch system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 2.5 𝘩𝘰𝘶𝘳𝘴 𝘰𝘧 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 & 𝘷𝘪𝘥𝘦𝘰 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴

Project mention: Full Stack 7-Steps MLOps Framework – Learn ML and MLOps by Building an ML System | news.ycombinator.com | 2023-09-26

• streamify

  4 474 0.0 Python

  A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• astronomer-cosmos

  8 457 9.4 Python

  Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code

  

• airflow-dbt

  1 379 0.0 Python

  Apache Airflow integration for dbt

  

• astro-sdk

  7 317 8.5 Python

  Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

  

Project mention: Orchestration: Thoughts on Dagster, Airflow and Prefect? | /r/dataengineering | 2023-06-01

Have you tried the Astro SDK? https://github.com/astronomer/astro-sdk

• airflow-chart

  1 267 7.4 Python

  A Helm chart to install Apache Airflow on Kubernetes

  

• airflow-docker

  2 223 3.0 Python

  Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j) (by coder2j)

• audiophile-e2e-pipeline

  3 170 0.0 Python

  Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard.

• airflowctl

  1 169 8.0 Python

  A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects

Project mention: A look at airflowctl, a tool to help developers manage Apache Airflow projects | dev.to | 2023-08-14

NOTE! I found a small issue in that when you run in background mode, it creates a file (.airflowctl/.background_process_ids) which contains the parent PID. The PID was always off, so I needed to manually edit this. I have created an issue here so if this happens to you, follow that.

• ask-astro

  2 155 8.9 Python

  An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer

  

Project mention: Show HN: Open-source implementation of A16Z’s LLM architecture | news.ycombinator.com | 2023-09-22

• airflow-testing-ci-workflow

  2 81 0.0 Python

  (project & tutorial) dag pipeline tests + ci/cd setup

• magniv-core

  1 75 0.0 Python

  Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.

  

• airflow-elt-blueprint

  1 42 2.9 Python

  A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.

• soopervisor

  3 42 5.0 Python

  ☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.

  

• dbt-cloud-plugin

  2 39 0.0 Python

  DBT Cloud Plugin for Airflow

• tfl-bikes-data-pipeline

  1 37 0.0 Python

  Processing TFL data for bike usage with Google Cloud Platform.

• typhoon-orchestrator

  14 29 0.0 Python

  Create elegant data pipelines and deploy to AWS Lambda or Airflow

  

• spotify-api

  2 29 3.0 Python

  Pipeline that extracts data from the Spotify API to build a more detailed version of Spotify Wrapped

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Python Airflow related posts

• Building in Public: Leveraging Tublian's AI Copilot for My Open Source Contributions

  1 project | dev.to | 12 Feb 2024

• Navigating Week Two: Insights and Experiences from My Tublian Internship Journey

  1 project | dev.to | 31 Dec 2023

• Airflow VS quix-streams - a user suggested alternative

  2 projects | 7 Dec 2023

• Best ETL Tools And Why To Choose

  1 project | /r/tactionsoftware | 11 Nov 2023

• Simplifying Data Transformation in Redshift: An Approach with DBT and Airflow

  2 projects | dev.to | 7 Nov 2023

• Share Your favorite python related software!

  1 project | /r/Python | 1 Oct 2023

• Show HN: Open-source implementation of A16Z’s LLM architecture

  1 project | news.ycombinator.com | 22 Sep 2023
• A note from our sponsor - SaaSHub
  www.saashub.com | 4 May 2024

  SaaSHub helps you find the best software and product alternatives Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!