Python Workflow engine

Open-source Python projects categorized as Workflow engine | Edit details

Top 9 Python Workflow engine Projects

  • GitHub repo Airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    Project mention: How to Serve Massive Computations Using Python Web Apps. | | 2021-11-23

    In this demo, we use the request itself as the trigger and begin computation immediately. But it may vary according to the nature of your application. Often, you might have to use a separate pipeline as well. In such scenarios, you may need technologies such as Apache Airflow or Prefect.

  • GitHub repo luigi

    Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

    Project mention: Noobie who is trying to use K8s needs confirmation to know if this is the way or he is overestimating Kubernetes. | | 2021-10-20
  • Scout APM

    Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.

  • GitHub repo Prefect

    The easiest way to automate your data

    Project mention: Prefect CLI Action | | 2021-11-21

    GitHub Action for running Prefect commands using the Prefect CLI.

  • GitHub repo Kedro

    A Python framework for creating reproducible, maintainable and modular data science code.

    Project mention: [Discussion] Applied machine learning implementation debate. Is OOP approach towards data preprocessing in python an overkill? | | 2021-11-03

    I'd focus more on understanding the issues in depth, before jumping to a solution. Otherwise, you would be adding hassle with some - bluntly speaking - opinionated and inflexible boilerplate code which not many people will like using. You mention some issues: non-obvious to understand code and hard to execute and replicate. Bad code which is not following engineering best practices (ideas from SOLID etc.) does not get better if you force the author to introduce certain classes. You can suggest some basics (e.g. common code formatter, meaningful variables names, short functions, no hard-coded values, ...), but I'm afraid you cannot educate non-engineers in a single day workshop. I would not focus on that at first. However, there is no excuse for writing bad code and then expecting others to fix. As you say, data engineering is part of data science skills, you are "junior" if you cannot write reproducible code. Being hard to execute and replicate is theoretically easy to fix. Force everyone to (at least hypothetically) submit their code into a testing environment where it will be automatically executed on a fresh machine. This will mean that at first they have to exactly specify all libraries that need to be installed. Second, they need to externalize all configuration - in particular data input and data output paths. Not a single value should be hard-coded in code! And finally they need a *single* command which can be run to execute the whole(!) pipeline. If they fail on any of these parts... they should try again. Work that does not pass this test is considered unfinished by the author. Basically you are introducing an automated, infallible test. Regarding your code, I'd really not try that direction. In particular even these few lines already look unclear and over-engineered. The csv format is already hard-coded into the code. If it changes to parquet you'd have to touch the code. The processing object has data paths fixed for which is no reason in a job which should take care of pure processing. Export data is also not something that a processing job should handle. And what if you have multiple input and output data? You would not have all these issues if you had kept to most simple solution to have a function `process(data1, data2, ...) -> result_data` where dataframes are passed in and out. It would also mean to have zero additional libraries or boilerplate. I highly doubt that a function `main_pipe(...)` will fix the malpractices some people may do. There are two small feature which are useful beyond a plain function though: automatically generating a visual DAG from the code and quick checking if input requirements are satisfied before heavy code is run. You can still put any mature DAG library on top, which probably already includes experience from a lot of developers. Not need to rewrite that. I'm not sure which one is best (metaflow, luigi, airflow, ... no idea), but many come with a lot of features. If you want a bit more scaffolding to easier understand foreign projects, you could look at but maybe that's already too much. Fix the "single command replication-from-scratch requirement" first.

  • GitHub repo viewflow

    Reusable workflow library for Django

    Project mention: Django library to edit business rules ? | | 2021-08-14 seems to be at least partially overlapping what you're looking for.

  • GitHub repo galaxy

    Data intensive science for everyone. (by galaxyproject)

    Project mention: Developed a new kind of dual extruder system on fully custom built 3D printer | | 2021-03-01
  • GitHub repo NIPY

    Workflows and interfaces for neuroimaging packages

  • Nanos

    Run Linux Software Faster and Safer than Linux with Unikernels.

  • GitHub repo redun

    Yet another redundant workflow engine

    Project mention: Insitro's new open source software uses DAGs. | | 2021-11-04
  • GitHub repo BPMN_RPA

    Robotic Process Automation in Windows and Linux by using BPMN diagrams.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-11-23.

Python Workflow engine related posts


What are some of the best open-source Workflow engine projects in Python? This list will help you:

Project Stars
1 Airflow 23,851
2 luigi 15,156
3 Prefect 7,821
4 Kedro 4,676
5 viewflow 2,124
6 galaxy 883
7 NIPY 595
8 redun 245
Find remote jobs at our new job board There are 34 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives