timely-dataflow
fluvio
Our great sponsors
timely-dataflow | fluvio | |
---|---|---|
11 | 26 | |
3,145 | 2,624 | |
1.1% | 4.6% | |
7.2 | 9.6 | |
19 days ago | 5 days ago | |
Rust | Rust | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
timely-dataflow
-
Readyset: A MySQL and Postgres wire-compatible caching layer
They have a bit about their technical foundation here[0].
Given that Readyset was co-founded by Jon Gjengset (but has apparently since departed the company), who authored the paper on Noria[1], I would assume that Readyset is the continuation of that research.
So it shares some roots with Materialize. They have a common conceptual ancestry in Naiad, where Materialize evolved out of timely-dataflow.
[0]: https://docs.readyset.io/concepts/streaming-dataflow
[1]: https://jon.thesquareplanet.com/papers/osdi18-noria.pdf
[2]: https://dl.acm.org/doi/10.1145/2517349.2522738
[3]: https://github.com/TimelyDataflow/timely-dataflow
-
Mandala: experiment data management as a built-in (Python) language feature
And systems like timely dataflow, https://github.com/TimelyDataflow/timely-dataflow
-
Arroyo: A distributed stream processing engine written in Rust
Project looks cool! Glad you open sourced it. It could use some comments in the code base to help contributors ;). I also like the datafusion usage, that is awesome. BTW I work on github.com/bytewax/bytewax, which is based on https://github.com/TimelyDataflow/timely-dataflow another Rust dataflow computation engine.
-
Rust MPI -- Will there ever be a fully oxidized implementation?
Just found this https://github.com/TimelyDataflow/timely-dataflow and my heart skipped a beat.
-
Streaming processing in Python using Timely Dataflow with Bytewax
Bytewax is a Python native binding to the Timely Dataflow library (written in Rust) for building highly scalable streaming (and batch) processing pipelines.
-
Alternative Kafka Integration Framework to Kafka Connect?
I am working on Bytewax, which is a Python stream processing framework built on Timely Dataflow. It is not exactly a Kafka integration framework because it is a more of a general stream processing framework, but might be interesting for you. We are focused on enabling people to more easily debug, containerize, parallelize and customize and less on enabling a declarative integration framework. It is still early days for us! And we are looking for feedback and ideas from the community.
-
[AskJS] JavaScript for data processing
We used to use a library called Pond.js, https://github.com/esnet/pond, but the reliance on Immutable.JS caused some performance pitfalls, so we wrote a system from scratch that deals with data in a batched streaming fashion. A lot of the concepts were borrowed from a Rust library called timely-dataflow, https://github.com/TimelyDataflow/timely-dataflow.
-
Dataflow: An Efficient Data Processing Library for Machine Learning
Though the name "Dataflow" might be an unfortunate name conflict with another Rust project: https://github.com/TimelyDataflow/timely-dataflow
-
Ask HN: Is there a way to subscribe to an SQL query for changes?
> In the simplest case, I'm talking about regular SQL non-materialized views which are essentially inlined.
I see that now -- makes sense!
> Wish we had some better database primitives to assemble rather than building everything on Postgres - its not ideal for a lot of things.
I'm curious to hear more about this! We agree that better primitives are required and that's why Materialize is written in Rust using using TimelyDataflow[1] and DifferentialDataflow[2] (both developed by Materialize co-founder Frank McSherry). The only relationship between Materialize and Postgres is that we are wire-compatible with Postgres and we don't share any code with Postgres nor do we have a dependence on it.
[1] https://github.com/TimelyDataflow/timely-dataflow
-
7 Real-Time Data Streaming Tools You Should Consider On Your Next Project
Under the hood, Materialize uses Timely Dataflow (TDF) as the stream-processing engine. This allows Materialize to take advantage of the distributed data-parallel compute engine. The great thing about using TDF is that it has been in open source development since 2014 and has since been battle-tested in production at large Fortune 1000-scale companies.
fluvio
- Ask HN: WebSocket Relay?
- XFaaS: Hyperscale and Low Cost Serverless Functions at Meta
-
Iggy.rs – building message streaming in Rust
I'm not quite sure how this compares to Kafka and fluvio [1], a Kafka competitor also written in Rust?
Is it more of a message queue like rabbitmq?
[1] https://www.fluvio.io/
- Fluvio: Open-source data streaming platform
- Show HN: Fluvio – Distributed stream processing system written in Rust and WASM
-
Thank you for checking out the Fluvio repo
Seeing signs of the hockey stick traffic on the Fluvio Open Source repo: https://github.com/infinyon/fluvio
Stopped the mobile notifications for the stargazer bot on discord and slack to stop the dopamine rush!!!
But thank you for checking us out.
- Opens Source Rust and WASM Alternative to Kafka and Flink
- Fluvio is a high-performance distributed data streaming platform in Rust
- RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
-
Advice: I am 36, and i want to transition from ETL developer to Rust
Look at open source project fluvio https://github.com/infinyon/fluvio and see how we’re building a modern bus that can do ETL streaming style :).
What are some alternatives?
noria - Fast web applications through dynamic, partially-stateful dataflow
nsq - A realtime distributed messaging platform
differential-datalog - DDlog is a programming language for incremental computation. It is well suited for writing programs that continuously update their output in response to input changes. A DDlog programmer does not write incremental algorithms; instead they specify the desired input-output mapping in a declarative manner.
datafuse - An elastic and reliable Cloud Warehouse, offers Blazing Fast Query and combines Elasticity, Simplicity, Low cost of the Cloud, built to make the Data Cloud easy [Moved to: https://github.com/datafuselabs/databend]
materialize - The data warehouse for operational workloads.
roapi - Create full-fledged APIs for slowly moving datasets without writing a single line of code.
bytewax - Python Stream Processing
sccache - Sccache is a ccache-like tool. It is used as a compiler wrapper and avoids compilation when possible. Sccache has the capability to utilize caching in remote storage environments, including various cloud storage options, or alternatively, in local storage.
realtime - Broadcast, Presence, and Postgres Changes via WebSockets
rust-blog - Educational blog posts for Rust beginners
differential-dataflow - An implementation of differential dataflow using timely dataflow on Rust.
cache - Cache dependencies and build outputs in GitHub Actions