Python Workflow engine

Open-source Python projects categorized as Workflow engine | Edit details

Top 7 Python Workflow engine Projects

  • GitHub repo Airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    Project mention: Install Airflow 2 on a Raspberry Pi (using Python 3.x) | dev.to | 2021-07-22

    Go to Airflow's github repo and download the airflow-webserver.service and the airflow-scheduler.service

  • GitHub repo luigi

    Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

    Project mention: Resources/tutorials to help me learn about ETL? | reddit.com/r/dataengineering | 2021-06-29
  • GitHub repo Prefect

    The easiest way to automate your data

    Project mention: Going from Jenkins to Airflow | reddit.com/r/dataengineering | 2021-07-10

    Im not sure what people's experience with PrefectPrefect is but i think its worth a look if you are considering Airflow to begin with. Granted I haven't used it at any large scale, I've found it easier to setup and maintain than Airflow.

  • GitHub repo Kedro

    A Python framework for creating reproducible, maintainable and modular data science code.

    Project mention: I Started Streaming on Twitch | dev.to | 2021-06-12

    It all started with kedro/issues/606, Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time consuming for me.

  • GitHub repo viewflow

    Reusable workflow library for Django

    Project mention: The Current State of Styling a Django Form/App | reddit.com/r/django | 2021-03-02

    I'm working on render django forms to Web Compoments, so any styling could be done on the client side.

  • GitHub repo galaxy

    Data intensive science for everyone. (by galaxyproject)

    Project mention: Developed a new kind of dual extruder system on fully custom built 3D printer | reddit.com/r/3Dprinting | 2021-03-01
  • GitHub repo NIPY

    Workflows and interfaces for neuroimaging packages

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-07-22.

Index

What are some of the best open-source Workflow engine projects in Python? This list will help you:

Project Stars
1 Airflow 22,472
2 luigi 14,804
3 Prefect 6,549
4 Kedro 4,103
5 viewflow 2,083
6 galaxy 853
7 NIPY 574