rslint VS lambdo

Author here. A few other Rust projects to note that I didn't mention in the original post I've since found:

- Boa (JS engine in Rust) – https://github.com/boa-dev/boa

- RSLint (JS/TS linter in Rust) – https://github.com/rslint/rslint

- Node version manager in Rust – https://github.com/Schniz/fnm

If you know of any other popular ones, let me know. I'm keeping a list :)

[4]: https://github.com/rslint/rslint

Having the possibility to update (query) output with new input data rather than process the whole input again even if the changes are very small is indeed a very useful feature. Assume that you have one huge input table and you computed the result consisting of a few rows. Now you add 1 record to the input. A traditional data processing system will again process all the input records while the differential system will update the existing output result.

There are the following difficulties in implementing such systems:

o (Small) changes in input have to be incrementally propagated to the output as updates rather than new results. This changes the paradigm of data processing because now any new operator has to be "update-aware"

o Only simple operators can be easily implemented as "update-aware". For more complex operators like aggregation or rolling aggregations, it is frequently not clear how it can be done conceptually

o Differential updates have to be propagated through a graph of operations (topology) which makes the task more difficult.

o Currently popular data processing approaches (SQL or map-reduce) were not designed for such a scenario so some adaptation might be needed

Another system where such an approach was implemented, called incremental evaluation, is Lambdo:

https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last!

Yet, this Python library relies on a different novel data processing paradigm where operations are applied to columns. Mathematically, it uses two types of operations: set operations and functions operations, as opposed to traditional approaches based on only set operations.

A new implementation is here:

https://github.com/asavinov/prosto - Functions matter! No join-groupby, No map-reduce.

Yet, incremental evaluation is implemented only for simple operations (calculated columns).

It will return the sum of all values in column A. For large tables it will take some time to compute the result. Now assume we append a new record and want to get the new result. The traditional approach is execute this query again. A better approach is to process this new record only by adding its value in A to the result of the previous query. It is important in (stateful) stream processing.

Something similar is implemented in these libraries which however rely on a different data processing conception (alternative to map-reduce):

https://github.com/asavinov/prosto - Functions matter! No join-groupby, No map-reduce.

https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last!

I find this project quite interesting because sklearn has a good general design including data transformations and it does make sense to provide compatible functionality for Go.

Feature engineering in general is a hot topic and especially if features are not simple hard-coded transformations but rather can be learned from data. For example, I developed a toolkit intended for combining feature engineering and ML:

    https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last!