redun
common-workflow-language
| redun | common-workflow-language | |
|---|---|---|
| 4 | 6 | |
| 486 | 1,441 | |
| 2.1% | 0.6% | |
| 7.5 | 1.1 | |
| about 2 months ago | 5 months ago | |
| Python | Common Workflow Language | |
| Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
redun
- Redun: Yet another redundant workflow engine
-
Nextflow: Data-Driven Computational Pipelines
I'm personally a huge fan of redun¹ for running computational pipelines. It's pure python, it's easy to learn/debug, it has automatic caching, retry, provenance logging, and a great integration with AWS Batch for running large jobs. I've been really impressed with how easy it is to run a job to completion that fans out to thousands of AWS spot instances at once.
I've used nextflow in the past, and I've found it to be much harder to use. Learning another DSL is annoying, documentation was sparse, I constantly ran into bugs, and it was hard to debug in general. I don't know how much it's changed over the past 3 years though.
¹https://github.com/insitro/redun
- Insitro's redun: Yet another redundant workflow engine
- Insitro's new open source software uses DAGs.
common-workflow-language
-
Nextflow: Data-Driven Computational Pipelines
https://www.commonwl.org/
https://github.com/common-workflow-language/common-workflow-...
- Common Workflow Language (CWL)
- Are there any good resources for building data pipelines?
- Common Workflow Language
-
Kestra - Open-Source Airflow Alternative
And we can add it to the ever-growing list of existing workflow frameworks; https://github.com/common-workflow-language/common-workflow-language/wiki/Existing-Workflow-systems
What are some alternatives?
Prefect - The easiest way to build, run, and monitor data pipelines at scale.
kestra - Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
nextflow - A DSL for data-driven computational pipelines
infinitic - Infinitic is a scalable workflow engine for distributed services. It shines particularly by making complex orchestration simple. It can be used to reliably orchestrate microservices, manage distributed transactions, operates data pipelines, builds user-facing automation, etc.
luigi - Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
toil - A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
cgpipe - cgpipe - minimum viable HPC pipeline
awesome-workflow-engines - A curated list of awesome open source workflow engines
common-workflow-
gh-action-pypi-publish - The blessed :octocat: GitHub Action, for publishing your :package: distribution files to PyPI: https://github.com/marketplace/actions/pypi-publish