SaaSHub helps you find the best software and product alternatives Learn more →
Top 5 pipeline-framework Open-Source Projects
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
FluentMediator
:twisted_rightwards_arrows: FluentMediator is an unobtrusive library that allows developers to build custom pipelines for Commands, Queries and Events.
-
SmartPipeline
A framework for rapid development of robust data pipelines following a simple design pattern
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
mediapipe/docs/solutions/pose.md at master · google/mediapipe · GitHub
> It's been a while since you can rerun/resume Nextflow pipelines
Yes, you can resume, but you need your whole upstream DAG to be present. Snakemake can rerun a job when only the dependencies of that job are present, which allows to neatly manage the disk usage, or archive an intermediate state of a project and rerun things from there.
> and yes, you can have dry runs in Nextflow
You have stubs, which really isn't the same thing.
> I have no idea what you're referring to with the 'arbitrary limit of 1000 parallel jobs' though
I was referring to this issue: https://github.com/nextflow-io/nextflow/issues/1871. Except, the discussion doesn't give the issue a full justice. Nextflow spans each job in a separate thread, and when it tries to span 1000+ condor jobs it die with a cryptic error message. The option of -Dnxf.pool.type=sync and -Dnxf.pool.maxThreads=N prevents the ability to resume and attempts to rerun the pipeline.
> As for deleting temporary files, there are features that allow you to do a few things related to that, and other features being implemented.
There are some hacks for this - but nothing I would feel safe to integrate into a production tool. They are implementing something - you're right - and it's been the case for several years now, so we'll see.
Snakemake has all that out of the box.
pipeline-framework related posts
-
HyperQueue: ergonomic HPC task executor written in Rust
-
Nextflow vs Snakemake
-
Deep Dive into Real-World Image Search Engine with Python
-
A quick tip on DataFrame.apply
-
Any good libraries for feature extraction?
-
A python framework for unstructured data processing
-
[P] Working on unstructured data?
-
A note from our sponsor - SaaSHub
www.saashub.com | 8 May 2024
Index
What are some of the best open-source pipeline-framework projects? This list will help you:
Project | Stars | |
---|---|---|
1 | mediapipe | 25,528 |
2 | nextflow | 2,544 |
3 | FluentMediator | 191 |
4 | machine | 142 |
5 | SmartPipeline | 23 |
Sponsored