timely-dataflow
realtime
Our great sponsors
timely-dataflow | realtime | |
---|---|---|
11 | 54 | |
3,125 | 6,418 | |
1.2% | 1.6% | |
7.2 | 9.0 | |
5 days ago | about 24 hours ago | |
Rust | Elixir | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
timely-dataflow
-
Readyset: A MySQL and Postgres wire-compatible caching layer
They have a bit about their technical foundation here[0].
Given that Readyset was co-founded by Jon Gjengset (but has apparently since departed the company), who authored the paper on Noria[1], I would assume that Readyset is the continuation of that research.
So it shares some roots with Materialize. They have a common conceptual ancestry in Naiad, where Materialize evolved out of timely-dataflow.
[0]: https://docs.readyset.io/concepts/streaming-dataflow
[1]: https://jon.thesquareplanet.com/papers/osdi18-noria.pdf
-
Mandala: experiment data management as a built-in (Python) language feature
And systems like timely dataflow, https://github.com/TimelyDataflow/timely-dataflow
-
Arroyo: A distributed stream processing engine written in Rust
Project looks cool! Glad you open sourced it. It could use some comments in the code base to help contributors ;). I also like the datafusion usage, that is awesome. BTW I work on github.com/bytewax/bytewax, which is based on https://github.com/TimelyDataflow/timely-dataflow another Rust dataflow computation engine.
-
Rust MPI -- Will there ever be a fully oxidized implementation?
Just found this https://github.com/TimelyDataflow/timely-dataflow and my heart skipped a beat.
-
Alternative Kafka Integration Framework to Kafka Connect?
I am working on Bytewax, which is a Python stream processing framework built on Timely Dataflow. It is not exactly a Kafka integration framework because it is a more of a general stream processing framework, but might be interesting for you. We are focused on enabling people to more easily debug, containerize, parallelize and customize and less on enabling a declarative integration framework. It is still early days for us! And we are looking for feedback and ideas from the community.
-
[AskJS] JavaScript for data processing
We used to use a library called Pond.js, https://github.com/esnet/pond, but the reliance on Immutable.JS caused some performance pitfalls, so we wrote a system from scratch that deals with data in a batched streaming fashion. A lot of the concepts were borrowed from a Rust library called timely-dataflow, https://github.com/TimelyDataflow/timely-dataflow.
-
Dataflow: An Efficient Data Processing Library for Machine Learning
Though the name "Dataflow" might be an unfortunate name conflict with another Rust project: https://github.com/TimelyDataflow/timely-dataflow
-
Ask HN: Is there a way to subscribe to an SQL query for changes?
> In the simplest case, I'm talking about regular SQL non-materialized views which are essentially inlined.
I see that now -- makes sense!
> Wish we had some better database primitives to assemble rather than building everything on Postgres - its not ideal for a lot of things.
I'm curious to hear more about this! We agree that better primitives are required and that's why Materialize is written in Rust using using TimelyDataflow[1] and DifferentialDataflow[2] (both developed by Materialize co-founder Frank McSherry). The only relationship between Materialize and Postgres is that we are wire-compatible with Postgres and we don't share any code with Postgres nor do we have a dependence on it.
-
7 Real-Time Data Streaming Tools You Should Consider On Your Next Project
Under the hood, Materialize uses Timely Dataflow (TDF) as the stream-processing engine. This allows Materialize to take advantage of the distributed data-parallel compute engine. The great thing about using TDF is that it has been in open source development since 2014 and has since been battle-tested in production at large Fortune 1000-scale companies.
- Why isn't differential dataflow more popular?
realtime
-
A Technical Dive into PostgreSQL's replication mechanisms
You can LISTEN/NOTIFY. Or you can use logical replication and a custom subscriber.[1] Supabase uses the latter.[2]
[1]: https://www.postgresql.org/docs/current/logical-replication....
-
Unpacking Elixir: Observability
We use :telemetry to collect usage data per tenant for Supabase Realtime.
We do this for rate limiting but it also makes it very easy for us to attach a listener (https://github.com/supabase/realtime/blob/main/lib/realtime/...) which ships these (per second) aggregates to BigQuery (via Logflare), which then the billing team can aggregate further to display and actually bill people with.
-
All the ways to capture changes in Postgres
Yo :D This is what Supabase Realtime does!
https://github.com/supabase/realtime
Spin up a Supabase database and then subscribe to changes with WebSockets.
You can play with it here once you have a db: https://realtime.supabase.com/inspector/new
-
Supabase Local Dev: migrations, branching, and observability
Every project is a Postgres database, wrapped in a suite of tools like Auth, Storage, Edge Functions, Realtime and Vectors, and encompassed by API middleware and logs.
-
Writing a chat application in Django 4.2 using async StreamingHttpResponse
Where can I learn more about this? I've been thinking of trying to integrate Supabase Realtime (https://github.com/supabase/realtime) into my Django app (without the rest of Supabase), but I'd also like to keep things even simpler if possible.
Also, what was the reason not to go with Gevent?
-
How to Listen to Database Changes Using Postgres Triggers in Elixir
I believe #2 was the main driver for the supabase team to build their real-time component: https://github.com/supabase/realtime
Background/announcement: https://supabase.com/blog/supabase-realtime-multiplayer-gene...
-
From Plex to Jellyfin Media Server
How does supabase not qualify as open source?
Their stack is primarily comprised of other independent open source projects. The one component that isn't is their "realtime" server that serves updates from postgres' WAL over websockets, but that is open sourced[0] under Apache 2.0. From my understanding the primary part that has not been open sourced is their database browser / web UI. There are plenty of alternative management tools for postgres though. As you can export your database what else would you need to ensure your portability and independence?
Granted they make their docs fairly opaque for trying to self host. Presumably to encourage you to just use their hosted service. Hosting open sourced projects seems like a very ecosystem friendly way of monetizing.
-
Supabase Subscriptions Just Got Easier
There is a few second delay when starting the subscriptions.
-
Finding Relationships Between Ruby’s Top Packages and Their Dependencies
Yes, some portion of their backend is Elixir/Phoenix: https://github.com/supabase/realtime
but most of their stack, frontend and backend, is Next.js, and not the Rails-type fullstack way: https://github.com/supabase/supabase
-
Streaming data in Postgres to 1M clients with GraphQL
They're all similar flavors of producing realtime results - which take similar, but different, methods to their approach.
My understanding (please feel free to correct me if I'm wrong):
- Supabase Realtime uses WAL.
- Hasura Streaming Subscriptions uses an append-only query (could also use WAL).
- Hasura Live Queries uses interval polling, refetching, and multiplexing.
- Supabase uses Postgres RLS for authorization, while Hasura uses an internal RLS system which composes queries (which allows for features like the multiplexing above).
- All 3 use websockets for their client communication.
Supabase Realtime
https://github.com/supabase/realtime#introduction
https://supabase.com/docs/guides/realtime
Hasura Subscriptions
https://github.com/hasura/graphql-engine/blob/master/archite...
https://github.com/hasura/graphql-engine/blob/master/archite...
What are some alternatives?
supabase - The open source Firebase alternative.
debezium - Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.
blockscout - Blockchain explorer for Ethereum based network and a tool for inspecting and analyzing EVM based blockchains.
Appwrite - Build like a team of hundreds_
litestream - Streaming replication for SQLite.
Hasura - Blazing fast, instant realtime GraphQL APIs on your DB with fine grained access control, also trigger webhooks on database events.
absurd-sql - sqlite3 in ur indexeddb (hopefully a better backend soon)
yugabyte-db - YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
tiptap - The headless rich text editor framework for web artisans.
multipass - Multipass orchestrates virtual Ubuntu instances
noria - Fast web applications through dynamic, partially-stateful dataflow
differential-datalog - DDlog is a programming language for incremental computation. It is well suited for writing programs that continuously update their output in response to input changes. A DDlog programmer does not write incremental algorithms; instead they specify the desired input-output mapping in a declarative manner.