Python Workflow engine

Open-source Python projects categorized as Workflow engine

Top 13 Python Workflow engine Projects

  • Airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    Project mention: .NET Modern Task Scheduler | reddit.com/r/dotnet | 2023-03-19

    A few years ago, I opened a GitHub issue with Microsoft telling them that I think the .NET ecosystem needs its own equivalent of Apache Airflow or Prefect. Fast forward 'til now, and I still don't think we have anything close to these frameworks.

  • luigi

    Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

    Project mention: In the context of Python what is a Bob Job? | reddit.com/r/learnpython | 2022-07-10

    Maybe if your use case is “smallish” and doesn’t require the whole studio suite you could check out apscheduler for doing python “tasks” on a schedule and luigi to build pipelines.

  • InfluxDB

    Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  • Prefect

    The easiest way to build, run, and monitor data pipelines at scale.

    Project mention: self hosted Alternative to easycron.com? | reddit.com/r/selfhosted | 2022-12-30
  • Kedro

    A Python framework for creating reproducible, maintainable and modular data science code.

    Project mention: Futuristic documentation systems in Python, part 1: aiming for more | dev.to | 2023-03-14

    Recently I started a position as Developer Advocate for Kedro, an opinionated data science framework, and one of the things we're doing is exploring what are the best open source tools we can use to create our documentation.

  • viewflow

    Reusable workflow library for Django

    Project mention: How to create a django ViewFlow process programmatically | reddit.com/r/codehunter | 2022-04-10

    I'm developing a web application to learn Django (python 3.4 & Django 1.6.10). The web app has complex and often updated workflows. I decided to integrate the Django-Viewflow library (https://github.com/viewflow/viewflow/) as it seems to be a very convenient way to handle workflows and not incorporate the workflow logic with the application models.

  • galaxy

    Data intensive science for everyone.

    Project mention: BIOINFORMATICS PROJECT | reddit.com/r/bioinformatics | 2022-10-16
  • NIPY

    Workflows and interfaces for neuroimaging packages

  • Sonar

    Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  • redun

    Yet another redundant workflow engine

    Project mention: Insitro's redun: Yet another redundant workflow engine | reddit.com/r/bioinformatics | 2022-08-24
  • jug

    Parallel programming with Python

  • flowsaber

    Dataflow based workflow framework

  • typhoon-orchestrator

    Create elegant data pipelines and deploy to AWS Lambda or Airflow

    Project mention: After Airflow. Where next for DE? | reddit.com/r/dataengineering | 2022-11-15
  • BPMN_RPA

    Robotic Process Automation in Windows and Linux by using Diagrams.net BPMN diagrams.

  • pyDag

    Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag

    Project mention: Data Engineering Projects for Beginners | dev.to | 2022-06-15

    Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-03-19.

Python Workflow engine related posts

Index

What are some of the best open-source Workflow engine projects in Python? This list will help you:

Project Stars
1 Airflow 29,478
2 luigi 16,401
3 Prefect 11,503
4 Kedro 8,179
5 viewflow 2,326
6 galaxy 1,076
7 NIPY 662
8 redun 393
9 jug 390
10 flowsaber 39
11 typhoon-orchestrator 29
12 BPMN_RPA 28
13 pyDag 18
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com