reflow VS lambdo

Reflow [1] is a similar attempt at a slightly different domain: bioinformatics and ETL pipelines. Reflow exposes a data model and programming model that reclaims programmability in these systems, and, by leaning on these abstractions, gives the runtime much more leeway to do interesting things. It unties the hands of the implementer.

[1] https://github.com/grailbio/reflow

https://github.com/grailbio/reflow is the closest that I know, as it has a design that resembles the Bazel build system.

It seems Reflow falls in this category:

https://github.com/grailbio/reflow

> Reflow thus allows scientists and engineers to write straightforward programs and then have them transparently executed in a cloud environment. Programs are automatically parallelized and distributed across multiple machines, and redundant computations (even across runs and users) are eliminated by its memoization cache. Reflow evaluates its programs incrementally: whenever the input data or program changes, only those outputs that depend on the changed data or code are recomputed.

It will return the sum of all values in column A. For large tables it will take some time to compute the result. Now assume we append a new record and want to get the new result. The traditional approach is execute this query again. A better approach is to process this new record only by adding its value in A to the result of the previous query. It is important in (stateful) stream processing.

Something similar is implemented in these libraries which however rely on a different data processing conception (alternative to map-reduce):

https://github.com/asavinov/prosto - Functions matter! No join-groupby, No map-reduce.

https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last!

I find this project quite interesting because sklearn has a good general design including data transformations and it does make sense to provide compatible functionality for Go.

Feature engineering in general is a hot topic and especially if features are not simple hard-coded transformations but rather can be learned from data. For example, I developed a toolkit intended for combining feature engineering and ML:

    https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last!