crux VS pg_bitemporal

Hi, a couple of my colleagues spent some time working on this integration with our open source database product (https://opencrux.com), and I'm curious to know - has anyone done similar things to connect Corda with a secondary off-the-shelf query engine?

For more details, see the release notes.

The context is that I work on on https://opencrux.com, which offers a bi-temporal Datalog query layer (as well as SQL) that more or less addresses the intersection of the two, since Datalog is great for expressing recursive queries.

I suppose another somewhat important distinction, once again performance related, is that graph databases will typically track index statistics to aid with query planning. For example, Crux uses stored knowledge of attribute-value cardinalities (recently via HyperLogLog) to optimise the join order of a query - this can make a big difference when attempting to traverse large graphs efficiently.

Agreed, recursive querying & bitemporal modelling in SQL are non-trivial problems, and the combination of the two is harder still. For an alternative perspective on tackling such problems I'd suggest looking at Datalog, which makes recursion a breeze, and a database with first-class bitemporality - both of which feature in https://opencrux.com (which I happen to work on :))

I work on Crux so can share a few details about our implementation of Datalog. The query is compiled into a kind of Worst-Case Optimal Join algorithm [0] which means that certain types of queries (e.g. cyclic graph-analytical queries, like counting triangles) are generally more efficient than what is possible with a non-WCOJ query execution strategy. However, the potency of this approach relies on the query planner calculating a good ordering of variables for the join order, and this is a hard problem in itself.

Crux is usually very competent at selecting a sensible variable ordering but when it makes a bad choice your query will take an unnecessary performance hit. The workaround for these situations is to break your query into smaller queries (since we don't wish to support any kind of hinting). Over the longer term we will be continuing to build more intelligent heuristics that make use of advanced population statistics. For instance we are about to merge a PR that uses HyperLogLog to inform attribute selectivity: https://github.com/juxt/crux/pull/1472

[0] https://cs.stanford.edu/people/chrismre/papers/paper49.Ngo.p...

I feel like i keep yelling the following, but bitemporal tables.

- https://aiven.io/blog/two-dimensional-time-with-bitemporal-d...

- https://github.com/scalegenius/pg_bitemporal

4 timestamps and some ugly queries.

People here may also be interested to see this analysis of the state of SQL:2011 "temporal table" feature adoption: https://illuminatedcomputing.com/posts/2019/08/sql2011-surve...

I don't think much has really changed since, and I'm not sure Postgres is any closer to addressing this natively (although there have been extensions, e.g. https://github.com/scalegenius/pg_bitemporal).

Something more sophisticated would be https://github.com/scalegenius/pg_bitemporal

Sure, I can appreciate that native support for a feature like this is nice.

As I understand it, most implementations (including another one for bitemporality[1]) involve either audit tables, as you mention, and/or additional support columns. It's as if the "now" representation is simply a narrow lens onto the underlying data.

That said, PostgreSQL encodes and has battle-tested decades of database functionality including an ecosystem around those, so I'd be a little wary of switching technology even if it does solve one individual problem thoroughly. Everything has to start somewhere, though.

[1] - https://github.com/scalegenius/pg_bitemporal