InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now. Learn more →
Top 17 Python Workflow engine Projects
-
Hi HN,
We've built an SDK for building DAGs / data pipelines with LLMs in Apache Airflow [1] using Pydantic AI [2] under the hood. I've seen success across the board with Airflow users building simple LLM workflows before moving on to "AI agents". In my experience, the noise around building agents means that people forget that there are other ways to get more immediate value out of LLMs.
Coupling Airflow for orchestration and Pydantic AI for LLM interactions has turned out to be a very pragmatic approach to building these workflows (and agents). Neither tool "gets in the way" of what you're trying to do. Airflow's been around for 10+ years and has a very well-built orchestration engine rich with everything you need to write production grade data pipelines, and Pydantic AI's been a refreshing take on working with LLMs.
Would love some feedback from this community!
[1] https://github.com/apache/airflow
-
InfluxDB
InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.
-
Project mention: Show HN: Flow – A Dynamic Task Engine for AI Agents Without DAG | news.ycombinator.com | 2024-12-02
- https://github.com/PrefectHQ/prefect
-
luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
-
-
Kedro
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
Project mention: 20 Open Source Tools I Recommend to Build, Share, and Run AI Projects | dev.to | 2024-11-13Kedro is an ML development framework that brings data science projects from pilot development to production by creating reproducible, maintainable, and modular data science code. Kedro has a data catalog for data handling, support pipeline building, and a standardized template for code maintainability and consistency to effectively do this. Its data catalog uses lightweight data connectors to manage and track datasets. This allows you to use the same pipeline to build multiple production-level codes across your system.
-
-
tracecat
Open source Tines / Splunk SOAR alternative. All-in-one automation platform (workflows, tables, cases) for security and IT teams.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
-
couler
Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
-
-
Project mention: Show HN: DBOS Python 1.0 – Lightweight Durable Workflows Built on Postgres | news.ycombinator.com | 2025-05-07
-
-
-
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python Workflow engine discussion
Python Workflow engine related posts
-
Personal Picks: Data Product News (March 19, 2025)
-
The DOJ Still Wants Google to Sell Off Chrome
-
AI Is Spamming Open Source Repos with Fake Issues
-
Enabling Apache Airflow to copy large S3 objects
-
Kedro – An open-source framework for data science code
-
New Apache Airflow Operators for Google Generative AI
-
Anyone Can Access Deleted and Private Repository Data on GitHub
-
A note from our sponsor - InfluxDB
www.influxdata.com | 13 May 2025
Index
What are some of the best open-source Workflow engine projects in Python? This list will help you:
| # | Project | Stars |
|---|---|---|
| 1 | Airflow | 39,973 |
| 2 | Prefect | 19,241 |
| 3 | luigi | 18,259 |
| 4 | dagster | 13,107 |
| 5 | Kedro | 10,322 |
| 6 | viewflow | 2,774 |
| 7 | tracecat | 2,631 |
| 8 | galaxy | 1,489 |
| 9 | couler | 936 |
| 10 | NIPY | 780 |
| 11 | dbos-python | 680 |
| 12 | redun | 544 |
| 13 | jug | 453 |
| 14 | BPMN_RPA | 42 |
| 15 | flowsaber | 41 |
| 16 | typhoon-orchestrator | 31 |
| 17 | pyDag | 24 |
Sponsored
InfluxDB – Built for High-Performance Time Series Workloads
InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.
www.influxdata.com