mz-raspberry-pi-temperature VS materialize

git clone https://github.com/bobbyiliev/mz-raspberry-pi-temperature.git

Thanks!

Definitely considering adding a server-mode with Postgres wire protocol compatibility.

It's tricky for the more dynamic/dataflow'y parts, as OctoSQL is able to give you a live updating output table (which Postgres wire protocol doesn't support), but I can go with a similar approach as Materialize[0] does for those use cases - creating a live-updating materialized view that you can query from.

That said, for now I'm still concentrating on the overall local usage experienced and ergonomics, there's still much to improve there.

[0]: https://materialize.com

hi! sujay from convex here. I remember reading about your "reverse query engine" when we were getting started last year and really liking that framing of the broadcast problem here.

as james mentions, we entirely re-run the javascript function whenever we detect any of its inputs change. incrementality at this layer would be very difficult, since we're dealing with a general purpose programming language. also, since we fully sandbox and determinize these javascript "queries," the majority of the cost is in accessing the database.

eventually, I'd like to explore "reverse query execution" on the boundary between javascript and the underlying data using an approach like differential dataflow [1]. the materialize folks [2] have made a lot of progress applying it for OLAP and readyset [3] is using similar techniques for OLTP.

[1] https://github.com/TimelyDataflow/differential-dataflow

[2] https://materialize.com/

[3] https://readyset.io/

https://materialize.com/ is billed to be that. That behavior is not trivial to implement.

Ha! Apparently I totally made that up—I meant https://materialize.com/

Yep, cache invalidation is hard because you can't just cache /users/1/networkstatus as a function of the request itself; its underlying value is a complex function of values fetched from possibly dozens of tables and services, any of which could change at any time without any immediately discernable connection to User 1.

OP's response to this, in another comment, is: "Now going back to your example with complicated dependencies. Maybe TTL is a better solution. With many dependencies, any changes from the dependency list might trigger invalidation. At some point, just doing TTL, would be simpler."

But what if you actually do want your dependency list to trigger invalidation?

https://materialize.com/ is the closest thing I know of to a "solution" for this - it lets you build nested, realtime materialized views on top of streaming and non-streaming data, all with SQL syntax and Postgres wire compatibility, that fully recalculate and recache their results whenever any input changes, no matter how deep (based on https://timelydataflow.github.io/timely-dataflow/ from Microsoft Research).

You could either use the outputs of such a system as a cache, or weave together a cache invalidation signaling system that sends a stream of keys that need to be evicted from cache at any time, directly into a Kafka topic. And then, of course, this could plug into Meta's cache invalidation consensus system. But it's remarkably disingenuous for Meta to pretend that the dependency side of this problem is trivial or can be solved by TTLs alone.

Materialize (licensed under BSL, not OSI-approved)

This is great, thank you so much! I just used it to clean up our workspace and was shocked from how many it found (I had already ran cargo-udeps in the past) https://github.com/MaterializeInc/materialize/pull/12099

The app is powered by Cloudflare Workers and Upstash Redis for storing data and Kafka for storing the click events along with Materialize for real-time data analytics.