timely-dataflow

A modular implementation of timely dataflow in Rust (by TimelyDataflow)

Timely-dataflow Alternatives

Similar projects and alternatives to timely-dataflow

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better timely-dataflow alternative or higher similarity.

timely-dataflow discussion

Log in or Post with

timely-dataflow reviews and mentions

Posts with mentions or reviews of timely-dataflow. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2024-02-21.
  • Readyset: A MySQL and Postgres wire-compatible caching layer
    5 projects | news.ycombinator.com | 21 Feb 2024
    They have a bit about their technical foundation here[0].

    Given that Readyset was co-founded by Jon Gjengset (but has apparently since departed the company), who authored the paper on Noria[1], I would assume that Readyset is the continuation of that research.

    So it shares some roots with Materialize. They have a common conceptual ancestry in Naiad, where Materialize evolved out of timely-dataflow.

    [0]: https://docs.readyset.io/concepts/streaming-dataflow

    [1]: https://jon.thesquareplanet.com/papers/osdi18-noria.pdf

    [2]: https://dl.acm.org/doi/10.1145/2517349.2522738

    [3]: https://github.com/TimelyDataflow/timely-dataflow

  • Mandala: experiment data management as a built-in (Python) language feature
    4 projects | /r/ProgrammingLanguages | 11 Apr 2023
    And systems like timely dataflow, https://github.com/TimelyDataflow/timely-dataflow
  • Arroyo: A distributed stream processing engine written in Rust
    3 projects | /r/rust | 4 Apr 2023
    Project looks cool! Glad you open sourced it. It could use some comments in the code base to help contributors ;). I also like the datafusion usage, that is awesome. BTW I work on github.com/bytewax/bytewax, which is based on https://github.com/TimelyDataflow/timely-dataflow another Rust dataflow computation engine.
  • Rust MPI -- Will there ever be a fully oxidized implementation?
    4 projects | /r/rust | 5 Mar 2023
    Just found this https://github.com/TimelyDataflow/timely-dataflow and my heart skipped a beat.
  • Streaming processing in Python using Timely Dataflow with Bytewax
    1 project | /r/Python | 9 Nov 2022
    Bytewax is a Python native binding to the Timely Dataflow library (written in Rust) for building highly scalable streaming (and batch) processing pipelines.
  • Alternative Kafka Integration Framework to Kafka Connect?
    3 projects | /r/apachekafka | 21 Jun 2022
    I am working on Bytewax, which is a Python stream processing framework built on Timely Dataflow. It is not exactly a Kafka integration framework because it is a more of a general stream processing framework, but might be interesting for you. We are focused on enabling people to more easily debug, containerize, parallelize and customize and less on enabling a declarative integration framework. It is still early days for us! And we are looking for feedback and ideas from the community.
  • [AskJS] JavaScript for data processing
    5 projects | /r/javascript | 27 May 2022
    We used to use a library called Pond.js, https://github.com/esnet/pond, but the reliance on Immutable.JS caused some performance pitfalls, so we wrote a system from scratch that deals with data in a batched streaming fashion. A lot of the concepts were borrowed from a Rust library called timely-dataflow, https://github.com/TimelyDataflow/timely-dataflow.
  • Dataflow: An Efficient Data Processing Library for Machine Learning
    2 projects | /r/rust | 17 Jan 2022
    Though the name "Dataflow" might be an unfortunate name conflict with another Rust project: https://github.com/TimelyDataflow/timely-dataflow
  • Ask HN: Is there a way to subscribe to an SQL query for changes?
    17 projects | news.ycombinator.com | 22 Apr 2021
    > In the simplest case, I'm talking about regular SQL non-materialized views which are essentially inlined.

    I see that now -- makes sense!

    > Wish we had some better database primitives to assemble rather than building everything on Postgres - its not ideal for a lot of things.

    I'm curious to hear more about this! We agree that better primitives are required and that's why Materialize is written in Rust using using TimelyDataflow[1] and DifferentialDataflow[2] (both developed by Materialize co-founder Frank McSherry). The only relationship between Materialize and Postgres is that we are wire-compatible with Postgres and we don't share any code with Postgres nor do we have a dependence on it.

    [1] https://github.com/TimelyDataflow/timely-dataflow

  • 7 Real-Time Data Streaming Tools You Should Consider On Your Next Project
    2 projects | dev.to | 20 Mar 2021
    Under the hood, Materialize uses Timely Dataflow (TDF) as the stream-processing engine. This allows Materialize to take advantage of the distributed data-parallel compute engine. The great thing about using TDF is that it has been in open source development since 2014 and has since been battle-tested in production at large Fortune 1000-scale companies.
  • A note from our sponsor - SaaSHub
    www.saashub.com | 16 Jul 2024
    SaaSHub helps you find the best software and product alternatives Learn more →

Stats

Basic timely-dataflow repo stats
11
3,221
7.5
7 days ago

Sponsored
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com