Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression. Learn more →
Top 13 Python Workflow engine Projects
-
A few years ago, I opened a GitHub issue with Microsoft telling them that I think the .NET ecosystem needs its own equivalent of Apache Airflow or Prefect. Fast forward 'til now, and I still don't think we have anything close to these frameworks.
-
luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Project mention: In the context of Python what is a Bob Job? | reddit.com/r/learnpython | 2022-07-10Maybe if your use case is “smallish” and doesn’t require the whole studio suite you could check out apscheduler for doing python “tasks” on a schedule and luigi to build pipelines.
-
InfluxDB
Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
-
-
Project mention: Futuristic documentation systems in Python, part 1: aiming for more | dev.to | 2023-03-14
Recently I started a position as Developer Advocate for Kedro, an opinionated data science framework, and one of the things we're doing is exploring what are the best open source tools we can use to create our documentation.
-
Project mention: How to create a django ViewFlow process programmatically | reddit.com/r/codehunter | 2022-04-10
I'm developing a web application to learn Django (python 3.4 & Django 1.6.10). The web app has complex and often updated workflows. I decided to integrate the Django-Viewflow library (https://github.com/viewflow/viewflow/) as it seems to be a very convenient way to handle workflows and not incorporate the workflow logic with the application models.
-
-
-
Sonar
Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.
-
Project mention: Insitro's redun: Yet another redundant workflow engine | reddit.com/r/bioinformatics | 2022-08-24
-
-
-
-
-
Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2023-03-19.
Python Workflow engine related posts
- Python SDK: The Release
- Why should I use Cloud Composer?
- How do you decide when to keep a project in a single python file vs break it up into multiple files?
- Mounting a Volume for Apache Airflow
- Airflow to Jenkins : How to set verifyssl False.
- Airflow Plugin Syncing
- Building a Data Lakehouse for Analyzing Elon Musk Tweets using MinIO, Apache Airflow, Apache Drill and Apache Superset
-
A note from our sponsor - InfluxDB
www.influxdata.com | 22 Mar 2023
Index
What are some of the best open-source Workflow engine projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | Airflow | 29,478 |
| 2 | luigi | 16,401 |
| 3 | Prefect | 11,503 |
| 4 | Kedro | 8,179 |
| 5 | viewflow | 2,326 |
| 6 | galaxy | 1,076 |
| 7 | NIPY | 662 |
| 8 | redun | 393 |
| 9 | jug | 390 |
| 10 | flowsaber | 39 |
| 11 | typhoon-orchestrator | 29 |
| 12 | BPMN_RPA | 28 |
| 13 | pyDag | 18 |
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com