blog VS differential-datalog

This reminds me of a great blog post by Frank McSherry (Materialize, timely dataflow, etc) talking about how using the right tools on a laptop could beat out a bunch of these JVM distributed querying tools because... data locality basically.

https://github.com/frankmcsherry/blog/blob/master/posts/2015...

Oh, cool!

And yeah, licenses can be challenging and frustrating, especially the first time you release a major project.

I am really super excited by the idea of embedded Datalog in Rust. I run into a lot of situations where I need something that fits in that awkward gap between SQL and Prolog. I want more expressiveness, better composability, and better graph support than SQL. But I also want finite-sized results that I can materialize in bounded time.

There has been some very neat work with incrementally-updated Datalog in the Rust community. For example, I think Datafrog is really neat: https://github.com/frankmcsherry/blog/blob/master/posts/2018... But it's great to see more neat projects in this space, so thank you.

They used to but Frank McSherry (author of differential dataflow) wrote them a specialized version without all the dataflow infrastructure [1]. It's part of the rust-lang nursery [2] now but hasn't been updated in a while, so I'm not sure what happened to it.

[1] https://github.com/frankmcsherry/blog/blob/master/posts/2018...

[2] https://github.com/rust-lang/datafrog

Importantly, this doesn't just use memoization (it actually avoids having to spend memory on that), but rather uses operators (nodes in the dataflow graph) that directly work with `(time, data, delta)` tuples. The `time` is a general lattice, so fairly flexible (e.g. for expressing loop nesting/recursive computations, but also for handling multiple input sources with their own timestamps), and the `delta` type is between a (potentially commutative) semigroup (don't be confused, they use addition as the group operation) and an abelian group. E.g. collections that are iteratively refined in loops often need an abelian `delta` type, while monoids (semigroup + explicit zero element) allow for efficient append-only computations [0].

[0]: https://github.com/frankmcsherry/blog/blob/master/posts/2019...

Hi,

> I wonder if it lives up to the hype.

We do think so! (disclaimer: I'm a co-founder at Feldera)

To give some more background: We are co-designing/trialing feldera with several industry/enterprise partners from different domains. Our core team also built differential datalog (https://github.com/vmware/differential-datalog) in the past. And while ddlog is used quite successfully in products today, we believe the many lessons we learned with ddlog will help us to build an even better continuous analytics platform. FYI our code is open-source at https://github.com/feldera/feldera if you'd like to try it out.

Also feel free to join our community slack channel (https://www.feldera.com/slack/) if you have more questions.

The relational model (and generally working at the level of sets/collections, instead of the level of individual values/objects) actually makes it easier to have this kind of incremental computation in a consistent way, I think.

There's a bunch of work being done on making relational systems work this way. Some interesting reading:

- https://www.scattered-thoughts.net/writing/an-opinionated-ma...

- https://materialize.com/ which is built on https://timelydataflow.github.io/differential-dataflow/, which has a lot of research behind it

- Which also can be a compilation target for Datalog: https://github.com/vmware/differential-datalog

- Some prototype work on building UI systems in exactly the way you describe using a relational approach: https://riffle.systems/essays/prelude/ (and HN discussion: https://news.ycombinator.com/item?id=30530120)

(There's a lot more too -- I have a hobby interest in this space, so I have a small collection of links)

Tutorial which I didn’t see linked in the README: https://github.com/vmware/differential-datalog/blob/master/d...

This is amazing!

Have you looked at differential-datalog? It's rust-based, maintained by VMWare, and has a very rich, well-typed Datalog language. differential-datalog is in-memory only right now, but could be ideal to integrate your graph as a datastore or disk spill cache.

https://github.com/vmware/differential-datalog

Sort of related, in my mind at least, is differential dataflow, e.g. https://github.com/vmware/differential-datalog

It’s fascinating to see so many different parties converging on Datalog for reactive apps & UI.

- There are several such talks at https://www.hytradboi.com/ (happening this Friday)

- Roam Research and its clones Athens, Logseq, use Datascript / ClojureScript https://github.com/tonsky/datascript

- differential-datalog isn’t an end-to-end system, but is highly optimized for quick reactivity https://github.com/vmware/differential-datalog

- Datalog UI is a Typescript port of some of differential-datalog’s ideas https://datalogui.dev/

Rust related https://github.com/vmware/differential-datalog

Actually, that makes me wonder whether or not differential datalog falls under that umbrella, and if it could be applied in the same way Compose is.