Is there an alternative for Airflow for running thousands of dynamic tasks?

This page summarizes the projects mentioned and recommended in the original post on /r/dataengineering
data-cleaning Architecture data-preparation Proposals Machine Learning

WorkOS
Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • Mage

    🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai

    • You could also check mage. https://github.com/mage-ai/mage-ai It is developed by former engineers of AirBnb too.

  • proposals

    Temporal proposals (by temporalio)

    • Check out temporal.io open source project. It was built at Uber for large scale business-level processes. So any data pipelines are low-rate use cases by definition.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts