awesome-pipeline
A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin (by pditommaso)
drake
An R-focused pipeline toolkit for reproducibility and high-performance computing (by ropensci)
| awesome-pipeline | drake | |
|---|---|---|
| 11 | 1 | |
| 6,354 | 1,342 | |
| 0.5% | 0.0% | |
| 6.4 | 6.3 | |
| 2 months ago | 5 months ago | |
| R | ||
| - | GNU General Public License v3.0 only |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
awesome-pipeline
Posts with mentions or reviews of awesome-pipeline.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2025-04-11.
-
Train Your Own LLM: A Deep Dive with Ruby
A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin - GitHub
-
Orchestration: Thoughts on Dagster, Airflow and Prefect?
There are a truly huge number of options in this space, see for example https://github.com/pditommaso/awesome-pipeline Many of them are very niche / half-baked / abandonware.
- Launch HN: DAGWorks – ML platform for data science teams
- Any good resources for R for bioinformatics
-
Alternatives to nextflow?
Hi everyone. So I've been using nextflow for about a month or so, having developed a few pipelines and I've found the debugging experience absolutely abysmal. Although nextflow has great observability with tower, and great community support with nf-core, the uninformative error messages is souring the experience for me. There are soooo many pipeline frameworks out there, but I'm wondering if anyone has come across one similar to nextflow in offering observability, a strong community behind it, multiple executors (container image based preferably) and an awesome debugging experience? I would favor a python based approach, but not sure snakemake is the one I'm looking for.
-
A General Workflow Engine
My answer is more from design/product point of view. If you mean code execution workflow management, then there are a bunch of packages listed in this awesome list.
- [Discussion] Applied machine learning implementation debate. Is OOP approach towards data preprocessing in python an overkill?
-
Experiences with workflow managers implemented in Haskell (funflow, porcupine, bioshake, ?)
There are a billion of them out there (https://github.com/pditommaso/awesome-pipeline), so the decision which one to choose is not exactly easy. Most of my colleagues rely on Nextflow and Snakemake, so I should consider these, but before I start to learn an entirely new language I wanted to explore the Haskell ecosystem for possible solutions. Strong typing should in theory be a perfect match for a pipeline manager. And having this in Haskell would simplify replacing some of my R code with Haskell eventually.
-
what do you think about airflow?
I found this list of other "awesome pipelines" https://github.com/pditommaso/awesome-pipeline
- Workflow Orchestration
drake
Posts with mentions or reviews of drake.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2021-05-02.
-
Your impression of {targets}? (r package)
The targets package is the official successor to Drake, and has the same primary author (Will Landau). He has explained why he created targets, which includes stronger guardrails for users and better UX.
What are some alternatives?
When comparing awesome-pipeline and drake you can also consider the following projects:
quokka - Making data lake work for time series
targets - Function-oriented Make-like declarative workflows for R
hamilton - Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
easystats - :milky_way: The R easystats-project
astro-sdk - Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
ncaahoopR - An R package for working with NCAA Basketball Play-by-Play Data