crux VS materialize

Hi, a couple of my colleagues spent some time working on this integration with our open source database product (https://opencrux.com), and I'm curious to know - has anyone done similar things to connect Corda with a secondary off-the-shelf query engine?

For more details, see the release notes.

The context is that I work on on https://opencrux.com, which offers a bi-temporal Datalog query layer (as well as SQL) that more or less addresses the intersection of the two, since Datalog is great for expressing recursive queries.

I suppose another somewhat important distinction, once again performance related, is that graph databases will typically track index statistics to aid with query planning. For example, Crux uses stored knowledge of attribute-value cardinalities (recently via HyperLogLog) to optimise the join order of a query - this can make a big difference when attempting to traverse large graphs efficiently.

Agreed, recursive querying & bitemporal modelling in SQL are non-trivial problems, and the combination of the two is harder still. For an alternative perspective on tackling such problems I'd suggest looking at Datalog, which makes recursion a breeze, and a database with first-class bitemporality - both of which feature in https://opencrux.com (which I happen to work on :))

I work on Crux so can share a few details about our implementation of Datalog. The query is compiled into a kind of Worst-Case Optimal Join algorithm [0] which means that certain types of queries (e.g. cyclic graph-analytical queries, like counting triangles) are generally more efficient than what is possible with a non-WCOJ query execution strategy. However, the potency of this approach relies on the query planner calculating a good ordering of variables for the join order, and this is a hard problem in itself.

Crux is usually very competent at selecting a sensible variable ordering but when it makes a bad choice your query will take an unnecessary performance hit. The workaround for these situations is to break your query into smaller queries (since we don't wish to support any kind of hinting). Over the longer term we will be continuing to build more intelligent heuristics that make use of advanced population statistics. For instance we are about to merge a PR that uses HyperLogLog to inform attribute selectivity: https://github.com/juxt/crux/pull/1472

[0] https://cs.stanford.edu/people/chrismre/papers/paper49.Ngo.p...

[2] https://materialize.com/

To fully leverage the data is the new oil concept, companies require a special database designed to manage vast amounts of data instantly. This need has led to different database forms, including NoSQL databases, vector databases, time-series databases, graph databases, in-memory databases, and in-memory data grids. Recent years have seen the rise of cloud-based streaming databases such as RisingWave, Materialize, DeltaStream, and TimePlus. While they each have distinct commercial and technical approaches, their overarching goal remains consistent: to offer users cloud-based streaming database services.

> Materialize no longer provide the latest code as an open-source software that you can download and try. It turned from a single binary design to cloud-only micro-service

Materialize CTO here. Just wanted to clarify that Materialize has always been source available, not OSS. Since our initial release in 2020, we've been licensed under the Business Source License (BSL), like MariaDB and CockroachDB. Under the BSL, each release does eventually transition to Apache 2.0, four years after its initial release.

Our core codebase is absolutely still publicly available on GitHub [0], and our developer guide for building and running Materialize on your own machine is still public [1].

It is true that we substantially rearchitected Materialize in 2022 to be more "cloud-native". Our new cloud offering offers horizontal scalability and fault tolerance—our two most requested features in the single-binary days. I wouldn't call the new architecture a microservices design though! There are only 2-3 services, each quite substantial, in the new architecture (loosely: a compute service, an orchestration service, and, soon, a load balancing service).

We do push folks to sign up for a free trial of our hosted cloud offering [2] these days, rather than trying to start off by running things locally, as we generally want folks' first impression of Materialize to be of the version that we support for production use cases. A all-in-one single machine Docker image does still exist, if you know where to look, but it's very much use-at-your-own-risk, and we don't recommend using it for anything serious, but it's there to support e.g. academic work that wants to evaluate Materialize's capabilities to incrementally maintain recursive SQL queries.

If folks have questions about Materialize, we've got a lively community Slack [3] where you can connect directly with our product and engineering teams.

[0]: https://github.com/MaterializeInc/materialize/tree/main

Some recent solutions to this problem include Differential Dataflow and Materialize. It would be neat if postgres adopted something similar for live-updating materialized views.

https://github.com/timelydataflow/differential-dataflow

https://materialize.com/

Materialize | Full-Time | NYC Office or Remote | https://materialize.com

Materialize is an Operational Data Warehouse: A cloud data warehouse with streaming internals, built for work that needs action on what’s happening right now. Keep the familiar SQL, keep the proven architecture of cloud warehouses but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date.

Materialize is the operational data warehouse built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI.

Senior/Staff Product Manager - https://grnh.se/69754ebf4us

Senior Frontend Engineer - https://grnh.se/7010bdb64us

===

Investors include Redpoint, Lightspeed and Kleiner Perkins.

Materialize | EM (Compute), Senior PM | New York, New York | https://materialize.com/

You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date.

That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI.

Engineering Manager, Compute - https://grnh.se/4e14099f4us

Senior Product Manager - https://grnh.se/587c36804us

VP of Marketing - https://grnh.se/9caac4b04us

How does it compare to https://materialize.com/ ?