arroyo
feldera
| arroyo | feldera | |
|---|---|---|
| 13 | 4 | |
| 3,326 | 268 | |
| 3.2% | 13.6% | |
| 9.6 | 9.9 | |
| 6 days ago | about 10 hours ago | |
| Rust | Rust | |
| Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
arroyo
- FLaNK AI Weekly 18 March 2024
- Arryo 0.8 released — streaming SQL engine
-
Query Engines: Push vs. Pull
Interesting - I looked into your code a bit. I found your window aggregation library [1]. You may be interested in looking into the Rust implementation of some of the research work I've been a part of [2].
In Flink, I believe the reason they need to implement their own backpressure system is that they multiplex TCP connections. That is, they have multiple logical streams flowing through a single TCP connection. If that's the case, you need to do some work to 1) detect which logical stream is the one that's blocking, and 2) don't block because other logical streams may be able to use the active TCP connection.
Thinking it through, I think what Flink's approach buys is not necessarily better performance, but better just a manageable number of connections. That is, imagine you have a process P1 with operators A, B and C. And then P2 has D, E, F. Now imagine that this is a shuffle, where A, B and C are fully connected to D, E and F. In my old system, you would have 9 TCP connections. In Flink, you will have 1.
[1] https://github.com/ArroyoSystems/arroyo/blob/master/arroyo-w...
- Arroyo
- Show HN: Arroyo – Write SQL on streaming data
- Release v0.3.0 · ArroyoSystems/arroyo - Stream Processing Engine
- Arroyo 0.2 released - Rust stream processing engine, now on Kubernetes
- Distributed stream processing engine written in Rust
- ArroyoSystems/arroyo: Arroyo is a distributed stream processing engine written in Rust
- Arroyo, a new open-source SQL stream processing engine written in Rust
feldera
-
We Built a Streaming SQL Engine
Very nice! Are you using the dbsp implementation from the paper authors for this (e.g., https://github.com/feldera/feldera/tree/main/crates/dbsp)?
-
Feldera – a more performant streaming database based on Z-sets
Hi,
> I wonder if it lives up to the hype.
We do think so! (disclaimer: I'm a co-founder at Feldera)
To give some more background: We are co-designing/trialing feldera with several industry/enterprise partners from different domains. Our core team also built differential datalog (https://github.com/vmware/differential-datalog) in the past. And while ddlog is used quite successfully in products today, we believe the many lessons we learned with ddlog will help us to build an even better continuous analytics platform. FYI our code is open-source at https://github.com/feldera/feldera if you'd like to try it out.
Also feel free to join our community slack channel (https://www.feldera.com/slack/) if you have more questions.
- FLaNK Stack Weekly for 14 Aug 2023
-
Show HN: Arroyo – Write SQL on streaming data
I'm a developer of DBSP. Our repo now lives here: https://github.com/feldera/dbsp/. And here is some more benchmarking data vs Flink and Beam:
What are some alternatives?
bytewax - Python Stream Processing
sql-cli-for-apache-flink-docker - SQL CLI for Apache Flink® via docker-compose
risingwave - SQL stream processing, analytics, and management. We decouple storage and compute to offer speedy bootstrapping, dynamic scaling, time-travel queries, and efficient joins.
incubator-fury - A blazingly fast multi-language serialization framework powered by JIT and zero-copy.
Benthos - Fancy stream processing made operationally mundane
llama2.c - Inference Llama 2 in one file of pure C
cli - Railway CLI
Llama-2-Onnx
timely-dataflow - A modular implementation of timely dataflow in Rust
databathing
sqlglot - Python SQL Parser and Transpiler
litellm - Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)