Python Workflow engine

Open-source Python projects categorized as Workflow engine

Top 17 Python Workflow engine Projects

Workflow engine
  1. Airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    Project mention: Airflow AI SDK to build simple LLM workflows | news.ycombinator.com | 2025-03-26

    Hi HN,

    We've built an SDK for building DAGs / data pipelines with LLMs in Apache Airflow [1] using Pydantic AI [2] under the hood. I've seen success across the board with Airflow users building simple LLM workflows before moving on to "AI agents". In my experience, the noise around building agents means that people forget that there are other ways to get more immediate value out of LLMs.

    Coupling Airflow for orchestration and Pydantic AI for LLM interactions has turned out to be a very pragmatic approach to building these workflows (and agents). Neither tool "gets in the way" of what you're trying to do. Airflow's been around for 10+ years and has a very well-built orchestration engine rich with everything you need to write production grade data pipelines, and Pydantic AI's been a refreshing take on working with LLMs.

    Would love some feedback from this community!

    [1] https://github.com/apache/airflow

  2. InfluxDB

    InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

    InfluxDB logo
  3. Prefect

    The easiest way to build, run, and monitor data pipelines at scale.

    Project mention: Show HN: Flow – A Dynamic Task Engine for AI Agents Without DAG | news.ycombinator.com | 2024-12-02

    - https://github.com/PrefectHQ/prefect

  4. luigi

    Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

  5. dagster

    An orchestration platform for the development, production, and observation of data assets.

    Project mention: Personal Picks: Data Product News (March 19, 2025) | dev.to | 2025-03-22
  6. Kedro

    Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

    Project mention: 20 Open Source Tools I Recommend to Build, Share, and Run AI Projects | dev.to | 2024-11-13

    Kedro is an ML development framework that brings data science projects from pilot development to production by creating reproducible, maintainable, and modular data science code. Kedro has a data catalog for data handling, support pipeline building, and a standardized template for code maintainability and consistency to effectively do this. Its data catalog uses lightweight data connectors to manage and track datasets. This allows you to use the same pipeline to build multiple production-level codes across your system.

  7. viewflow

    Reusable workflow library for Django

  8. tracecat

    Open source Tines / Splunk SOAR alternative. All-in-one automation platform (workflows, tables, cases) for security and IT teams.

    Project mention: N8n – Open-source Zapier alternative | news.ycombinator.com | 2025-05-03
  9. SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
  10. galaxy

    Data intensive science for everyone.

  11. couler

    Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

  12. NIPY

    Workflows and interfaces for neuroimaging packages

  13. dbos-python

    Lightweight Durable Python Workflows

    Project mention: Show HN: DBOS Python 1.0 – Lightweight Durable Workflows Built on Postgres | news.ycombinator.com | 2025-05-07
  14. redun

    Yet another redundant workflow engine

  15. jug

    Parallel programming with Python

  16. BPMN_RPA

    Robotic Process Automation in Windows and Linux by using Diagrams.net BPMN diagrams.

  17. flowsaber

    Dataflow based workflow framework

  18. typhoon-orchestrator

    Create elegant data pipelines and deploy to AWS Lambda or Airflow

  19. pyDag

    Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag

  20. SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python Workflow engine discussion

Log in or Post with

Python Workflow engine related posts

  • Personal Picks: Data Product News (March 19, 2025)

    1 project | dev.to | 22 Mar 2025
  • The DOJ Still Wants Google to Sell Off Chrome

    4 projects | news.ycombinator.com | 8 Mar 2025
  • AI Is Spamming Open Source Repos with Fake Issues

    1 project | news.ycombinator.com | 5 Feb 2025
  • Enabling Apache Airflow to copy large S3 objects

    2 projects | dev.to | 26 Aug 2024
  • Kedro – An open-source framework for data science code

    1 project | news.ycombinator.com | 17 Aug 2024
  • New Apache Airflow Operators for Google Generative AI

    1 project | news.ycombinator.com | 12 Aug 2024
  • Anyone Can Access Deleted and Private Repository Data on GitHub

    7 projects | news.ycombinator.com | 24 Jul 2024
  • A note from our sponsor - InfluxDB
    www.influxdata.com | 13 May 2025
    InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now. Learn more →

Index

What are some of the best open-source Workflow engine projects in Python? This list will help you:

# Project Stars
1 Airflow 39,973
2 Prefect 19,241
3 luigi 18,259
4 dagster 13,107
5 Kedro 10,322
6 viewflow 2,774
7 tracecat 2,631
8 galaxy 1,489
9 couler 936
10 NIPY 780
11 dbos-python 680
12 redun 544
13 jug 453
14 BPMN_RPA 42
15 flowsaber 41
16 typhoon-orchestrator 31
17 pyDag 24

Sponsored
InfluxDB – Built for High-Performance Time Series Workloads
InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.
www.influxdata.com

Did you know that Python is
the 2nd most popular programming language
based on number of references?