cli VS arroyo

We'll first install the CLI on our local system, for that the guide is quite limited in a way for a few options to choose from like npm, shell, and scoop. For me, the shell was the way to go, but it had a few issues with permission, so I made a few changes in the install.sh script ran on my machine and it worked fine.

Interesting - I looked into your code a bit. I found your window aggregation library [1]. You may be interested in looking into the Rust implementation of some of the research work I've been a part of [2].

In Flink, I believe the reason they need to implement their own backpressure system is that they multiplex TCP connections. That is, they have multiple logical streams flowing through a single TCP connection. If that's the case, you need to do some work to 1) detect which logical stream is the one that's blocking, and 2) don't block because other logical streams may be able to use the active TCP connection.

Thinking it through, I think what Flink's approach buys is not necessarily better performance, but better just a manageable number of connections. That is, imagine you have a process P1 with operators A, B and C. And then P2 has D, E, F. Now imagine that this is a shuffle, where A, B and C are fully connected to D, E and F. In my old system, you would have 9 TCP connections. In Flink, you will have 1.

[1] https://github.com/ArroyoSystems/arroyo/blob/master/arroyo-w...