crux VS xforms

Hi, a couple of my colleagues spent some time working on this integration with our open source database product (https://opencrux.com), and I'm curious to know - has anyone done similar things to connect Corda with a secondary off-the-shelf query engine?

For more details, see the release notes.

The context is that I work on on https://opencrux.com, which offers a bi-temporal Datalog query layer (as well as SQL) that more or less addresses the intersection of the two, since Datalog is great for expressing recursive queries.

I suppose another somewhat important distinction, once again performance related, is that graph databases will typically track index statistics to aid with query planning. For example, Crux uses stored knowledge of attribute-value cardinalities (recently via HyperLogLog) to optimise the join order of a query - this can make a big difference when attempting to traverse large graphs efficiently.

Agreed, recursive querying & bitemporal modelling in SQL are non-trivial problems, and the combination of the two is harder still. For an alternative perspective on tackling such problems I'd suggest looking at Datalog, which makes recursion a breeze, and a database with first-class bitemporality - both of which feature in https://opencrux.com (which I happen to work on :))

I work on Crux so can share a few details about our implementation of Datalog. The query is compiled into a kind of Worst-Case Optimal Join algorithm [0] which means that certain types of queries (e.g. cyclic graph-analytical queries, like counting triangles) are generally more efficient than what is possible with a non-WCOJ query execution strategy. However, the potency of this approach relies on the query planner calculating a good ordering of variables for the join order, and this is a hard problem in itself.

Crux is usually very competent at selecting a sensible variable ordering but when it makes a bad choice your query will take an unnecessary performance hit. The workaround for these situations is to break your query into smaller queries (since we don't wish to support any kind of hinting). Over the longer term we will be continuing to build more intelligent heuristics that make use of advanced population statistics. For instance we are about to merge a PR that uses HyperLogLog to inform attribute selectivity: https://github.com/juxt/crux/pull/1472

[0] https://cs.stanford.edu/people/chrismre/papers/paper49.Ngo.p...

Maybe https://github.com/cgrand/xforms The for transducer might help, just as the for comprehension helps unpack and map/filter nested stuff.

cgrand/xforms is a very useful hidden gem, if you like transducers/eager evaluation/solving map-vals without meander/specter.

* It's fast enough for 99% of apps out of the box. It's fast enough for 99.99% of the apps with minimal tuning.

* Yes, if your project is very big and macro heavy, it can take some time, but startup times have improved. In any case, I BARELY need to restart my development JVM. I have one currently running that I haven't restarted for 1 week+.

* Depending on what's your cup of tea, there's emacs/CIDER or IntelliJ/Cursive. They both work well. IntelliJ/Cursive is an excellent IDE combination. I use it every day.

* Java interop is very straightforward, not sure what you mean. Sure your code might not be all pure anymore, but that's the price for solving actual problems.

* Good java libraries have wrappers. A ton of original Clojure libraries as well. https://github.com/cgrand/xforms for example allows you to easily do things that I can't even imagine doing in an imperative language.

* Static vs dynamic typing: don't want to get into that.

* "Clojurescript isn't the same language". I use both Clojure and ClojureScript every day and as far as Clojure-only code is concerned, it works in both languages 99.99% of the time. One case you can encounter issues is if you do something host-specific, like dealing with numbers. That's by design. Clojure embraces each host, does not try to reinvent it. When you just use pure Clojure data structure manipulation, it works the same across both languages and works like magic.