Lessons Learned from Running Apache Airflow at Scale

• windmill

  86 8,518 10.0 Svelte

  Open-source developer platform to turn scripts into workflows and UIs. Fastest workflow engine (5x vs Airflow). Open-source alternative to Airplane and Retool.

  

shameless plug: I am building such system where the modules are code (typescript-deno or python) but the orchestration is no code (flows). It is fully OSS: https://github.com/windmill-labs/windmill

• orchest

  44 4,020 4.5 TypeScript

  Build data pipelines, the easy way 🛠️

  

We kept hearing this from our users. We’ve just released our k8s operator based deployment of Orchest that should give you a good experience running an orchestration tool on k8s without much trouble.

https://github.com/orchest/orchest

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• cronitor-airflow

  1 2 10.0 Python

  Cronitor integration for Airflow

  

Is anybody out there doing anything interesting with Airflow monitoring?

At my startup Cronitor we have an Airflow sdk* that makes it pretty easy to provision monitoring for each DAG, but essentially we are only monitoring that a DAG started on time and the total time taken.

* https://github.com/cronitorio/cronitor-airflow

• states-language-cadence

  1 63 10.0 Go

  States Language on Cadence

  

https://github.com/checkr/states-language-cadence allows you to define workflows in states language over cadence.

• ploomber

  121 3,369 7.8 Python

  The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

  

When it comes to scale and DS work I'd use the ploomber open-source (https://github.com/ploomber/ploomber). It allows an easy transition between dev and production, incrementally building the DAG so you avoid expensive compute time and costs. It's easier to maintain and integrates seamlessly with Airflow, generating the DAGs for you.

• dbt-core

  86 8,881 9.7 Python

  dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

  

dbt has just opened a serious conversation about supporting Python models. I'm sure they'd value your viewpoint! https://github.com/dbt-labs/dbt-core/discussions/5261

• luigi

  14 17,292 6.4 Python

  Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

  

What are you trying to do? Distributed scheduler with a single instance? No database? Are you sure you don't just mean "a scheduler" ala Luigi? https://github.com/spotify/luigi

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• magniv-core

  1 73 0.0 Python

  Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.

  

We at magniv.io are building an alternative.

Our core is open source https://github.com/MagnivOrg/magniv-core

We can set you up with our hosted if you would like to poke around!

• proposals

  60 63 4.0

  Temporal proposals (by temporalio)

  

You're probably thinking of Temporal (https://temporal.io/), which is a fork of the Cadence project originally developed at Uber.

• toil

  2 869 9.2 Python

  A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

  

• stepwise

  2 37 3.8 Clojure

  Clojure AWS Step Functions library

  

I feel you. That's why we wrote a little library on top of SFN so that we can program SFN with Clojure instead of YAML https://github.com/Motiva-AI/stepwise. Application code sits with SFN definition and SFN Tasks are automatically integrated as polling Activities from Clojure code.

Thoughtworks made a case for this distinction in https://martinfowler.com/articles/cant-buy-integration.html#...

• direktiv

  13 461 10.0 TypeScript

  Serverless Container Orchestration

  

So being completely transparent, we're the creators of Direktiv (https://github.com/direktiv/direktiv). We're genuinely curious to have users who have previously used Airflow and other DAGs (mentioned in here is Argo workflows) try Direktiv and give us more feedback.

- direktiv runs containers as part of workflows from any compliant container registry, passing JSON structured data between workflow states.

Suggest a related project