differential-dataflow
materialize
differential-dataflow | materialize | |
---|---|---|
14 | 117 | |
2,473 | 5,580 | |
0.8% | 0.4% | |
8.3 | 10.0 | |
5 days ago | 1 day ago | |
Rust | Rust | |
MIT License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
differential-dataflow
-
We Built a Streaming SQL Engine
Some recent solutions to this problem include Differential Dataflow and Materialize. It would be neat if postgres adopted something similar for live-updating materialized views.
https://github.com/timelydataflow/differential-dataflow
https://materialize.com/
-
Hydroflow: Dataflow Runtime in Rust
I'm looking for this but can't find it, how does this project compare to differential dataflow?
As a sibling commenter mentioned, it's built on timely dataflow (which is lower-level), but that already has differential dataflow[0] built on top of it by the same authors.
How do they differ?
[0]: https://github.com/TimelyDataflow/differential-dataflow
- Using Rust to write a Data Pipeline. Thoughts. Musings.
- PlanetScale Boost
- Program Synthesis is Possible (2018)
-
Convex vs. Firebase
hi! sujay from convex here. I remember reading about your "reverse query engine" when we were getting started last year and really liking that framing of the broadcast problem here.
as james mentions, we entirely re-run the javascript function whenever we detect any of its inputs change. incrementality at this layer would be very difficult, since we're dealing with a general purpose programming language. also, since we fully sandbox and determinize these javascript "queries," the majority of the cost is in accessing the database.
eventually, I'd like to explore "reverse query execution" on the boundary between javascript and the underlying data using an approach like differential dataflow [1]. the materialize folks [2] have made a lot of progress applying it for OLAP and readyset [3] is using similar techniques for OLTP.
[1] https://github.com/TimelyDataflow/differential-dataflow
[2] https://materialize.com/
[3] https://readyset.io/
-
Announcing avalanche 0.1, a React- and Svelte-inspired GUI library
differential dataflow which is used to power materialize db
-
Differential Datalog
It's partially inspired by Linq, so the similarity you see is expected.
It's not really arbitrary structures so much, though you're mostly free in what record type you use in a relation (structs and tagged enums are typical, though).
The incremental part is that you can feed it changes to the input (additions/retractions of facts) and get changes to the outputs back with low latency (you can alternatively just use it to keep an index up-to-date, where you can quickly look up based on a key (like a materialized view in SQL)).
This [0] section in the readme of the underlying incremental dataflow framework may help get the concept across, but feel free to follow up if you're still not seeing the incrementality.
[0]: https://github.com/TimelyDataflow/differential-dataflow#an-e...
- Dbt and Materialize
- Materialized view questions
materialize
-
Ask HN: How Can I Make My Front End React to Database Changes in Real-Time?
[2] https://materialize.com/
-
Choosing Between a Streaming Database and a Stream Processing Framework in Python
To fully leverage the data is the new oil concept, companies require a special database designed to manage vast amounts of data instantly. This need has led to different database forms, including NoSQL databases, vector databases, time-series databases, graph databases, in-memory databases, and in-memory data grids. Recent years have seen the rise of cloud-based streaming databases such as RisingWave, Materialize, DeltaStream, and TimePlus. While they each have distinct commercial and technical approaches, their overarching goal remains consistent: to offer users cloud-based streaming database services.
-
Proton, a fast and lightweight alternative to Apache Flink
> Materialize no longer provide the latest code as an open-source software that you can download and try. It turned from a single binary design to cloud-only micro-service
Materialize CTO here. Just wanted to clarify that Materialize has always been source available, not OSS. Since our initial release in 2020, we've been licensed under the Business Source License (BSL), like MariaDB and CockroachDB. Under the BSL, each release does eventually transition to Apache 2.0, four years after its initial release.
Our core codebase is absolutely still publicly available on GitHub [0], and our developer guide for building and running Materialize on your own machine is still public [1].
It is true that we substantially rearchitected Materialize in 2022 to be more "cloud-native". Our new cloud offering offers horizontal scalability and fault tolerance—our two most requested features in the single-binary days. I wouldn't call the new architecture a microservices design though! There are only 2-3 services, each quite substantial, in the new architecture (loosely: a compute service, an orchestration service, and, soon, a load balancing service).
We do push folks to sign up for a free trial of our hosted cloud offering [2] these days, rather than trying to start off by running things locally, as we generally want folks' first impression of Materialize to be of the version that we support for production use cases. A all-in-one single machine Docker image does still exist, if you know where to look, but it's very much use-at-your-own-risk, and we don't recommend using it for anything serious, but it's there to support e.g. academic work that wants to evaluate Materialize's capabilities to incrementally maintain recursive SQL queries.
If folks have questions about Materialize, we've got a lively community Slack [3] where you can connect directly with our product and engineering teams.
[0]: https://github.com/MaterializeInc/materialize/tree/main
- What I Talk About When I Talk About Query Optimizer (Part 1): IR Design
-
We Built a Streaming SQL Engine
Some recent solutions to this problem include Differential Dataflow and Materialize. It would be neat if postgres adopted something similar for live-updating materialized views.
https://github.com/timelydataflow/differential-dataflow
https://materialize.com/
-
Ask HN: Who is hiring? (October 2023)
Materialize | Full-Time | NYC Office or Remote | https://materialize.com
Materialize is an Operational Data Warehouse: A cloud data warehouse with streaming internals, built for work that needs action on what’s happening right now. Keep the familiar SQL, keep the proven architecture of cloud warehouses but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date.
Materialize is the operational data warehouse built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI.
Senior/Staff Product Manager - https://grnh.se/69754ebf4us
Senior Frontend Engineer - https://grnh.se/7010bdb64us
===
Investors include Redpoint, Lightspeed and Kleiner Perkins.
-
Ask HN: Who is hiring? (June 2023)
Materialize | EM (Compute), Senior PM | New York, New York | https://materialize.com/
You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date.
That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI.
Engineering Manager, Compute - https://grnh.se/4e14099f4us
Senior Product Manager - https://grnh.se/587c36804us
VP of Marketing - https://grnh.se/9caac4b04us
- What are your favorite tools or components in the Kafka ecosystem?
- Ask HN: Who is hiring? (May 2023)
-
Dozer: A scalable Real-Time Data APIs backend written in Rust
How does it compare to https://materialize.com/ ?
What are some alternatives?
ballista - Distributed compute platform implemented in Rust, and powered by Apache Arrow.
ClickHouse - ClickHouse® is a free analytics DBMS for big data
reflow - A language and runtime for distributed, incremental data processing in the cloud
risingwave - Cloud-native SQL stream processing, analytics, and management. KsqlDB and Apache Flink alternative. 🚀 10x more productive. 🚀 10x more cost-efficient.
differential-datalog - DDlog is a programming language for incremental computation. It is well suited for writing programs that continuously update their output in response to input changes. A DDlog programmer does not write incremental algorithms; instead they specify the desired input-output mapping in a declarative manner.
openpilot - openpilot is an open source driver assistance system. openpilot performs the functions of Automated Lane Centering and Adaptive Cruise Control for 250+ supported car makes and models.
timely-dataflow - A modular implementation of timely dataflow in Rust
rust-kafka-101 - Getting started with Rust and Kafka
clj-3df - Clojure(Script) client for Declarative Dataflow.
dbt-expectations - Port(ish) of Great Expectations to dbt test macros
rslint - A (WIP) Extremely fast JavaScript and TypeScript linter and Rust crate
scryer-prolog - A modern Prolog implementation written mostly in Rust.