flowbase VS scipipe

I see a lot of potential in pipeline concurrency, as seen in dataflow (DF) and flow-based programming (FBP). That is, modeling computation as pipelines where one component sends data to the next component via message passing. As long as there is enough data it will be possible for multiple components in the chain to work concurrently.

The benefits are that no other synchronization is needed than the data sent between processes, and race conditions are ruled out as long as only one process is allowed to process a data item at a time (this is the rule in FBP).

The main blockers I think is that it requires quite a rethink of the architecture of software. I see this rethink happening in larger, especially distributed systems, which are modeled a lot around these principles already, using systems such as Kafka and message queues to communicate, which more or less forces people to model computations around the data flow.

I think the same could happen inside monolithic applications too, with the right tooling. The concurrency primitives in Go are superbly suited to this in my experience, given that you work with the right paradigm, which I've been writing about before [1, 2], and started making a micro-unframework for [3] (though the latter one will be possible to make so much nicer after we get generics in Go).

But then, I also think there are some lessons to be learned about the right granularity for processes and data in the pipeline. Due to the overhead of message passing, it will not make sense performance-wise to use dataflow for the very finest-grain data.

Perhaps this in a sense parallels what we see with distributed computing, where there is a certain breaking point before which it isn't really worth it to go with distributed computing, because of all the overhead, both performance-wise and complexity-wise.

[1] https://blog.gopheracademy.com/composable-pipelines-pattern/

[2] https://blog.gopheracademy.com/advent-2015/composable-pipeli...

[3] https://flowbase.org

https://scipipe.org - A pipeline tool for shell commands by a declarative flow-based API in Go

Github link: https://github.com/scipipe/scipipe

There are many pipeline tools for shell commands, but a majority has one or more limitations in their API which makes certain complex pipelines impossible or really hard to write.

We were pushing the limits of all the tools we tried, so developed our own, and implemented it in Go, with a declarative API for defining the data flow dependencies, instead of inventing yet another DSL. This has allowed us great flexibility in developing also complex pipelines, e.g. combining parameter sweeps nested with cross-validation implemented as workflow constructs.

SciPipe is also unique in providing an audit report for every single output of the workflow, in a structured JSON format. A helper tool allows converting these reports to either an HTML report, a PDF, or a Bash script that will generate the one accompanying output file from scratch.

An extra cool things is that, because the audit reports live alongside output files, if you run a scipipe workflow that uses files generated by another scipipe workflow, it will pick up also all the history for the input files generated by this earlier workflow, meaning that you get a 100% complete audit report, even if your analysis spans multiple workflows!