crux VS malli

Hi, a couple of my colleagues spent some time working on this integration with our open source database product (https://opencrux.com), and I'm curious to know - has anyone done similar things to connect Corda with a secondary off-the-shelf query engine?

For more details, see the release notes.

The context is that I work on on https://opencrux.com, which offers a bi-temporal Datalog query layer (as well as SQL) that more or less addresses the intersection of the two, since Datalog is great for expressing recursive queries.

I suppose another somewhat important distinction, once again performance related, is that graph databases will typically track index statistics to aid with query planning. For example, Crux uses stored knowledge of attribute-value cardinalities (recently via HyperLogLog) to optimise the join order of a query - this can make a big difference when attempting to traverse large graphs efficiently.

Agreed, recursive querying & bitemporal modelling in SQL are non-trivial problems, and the combination of the two is harder still. For an alternative perspective on tackling such problems I'd suggest looking at Datalog, which makes recursion a breeze, and a database with first-class bitemporality - both of which feature in https://opencrux.com (which I happen to work on :))

I work on Crux so can share a few details about our implementation of Datalog. The query is compiled into a kind of Worst-Case Optimal Join algorithm [0] which means that certain types of queries (e.g. cyclic graph-analytical queries, like counting triangles) are generally more efficient than what is possible with a non-WCOJ query execution strategy. However, the potency of this approach relies on the query planner calculating a good ordering of variables for the join order, and this is a hard problem in itself.

Crux is usually very competent at selecting a sensible variable ordering but when it makes a bad choice your query will take an unnecessary performance hit. The workaround for these situations is to break your query into smaller queries (since we don't wish to support any kind of hinting). Over the longer term we will be continuing to build more intelligent heuristics that make use of advanced population statistics. For instance we are about to merge a PR that uses HyperLogLog to inform attribute selectivity: https://github.com/juxt/crux/pull/1472

[0] https://cs.stanford.edu/people/chrismre/papers/paper49.Ngo.p...

Clojure's lazy sequences by default are wonderful ergonomically, but it provides many ways to use strict evaluation if you want to. They aren't really a hassle either. I've been doing Clojure for the last few years and have a few grievances, but overall it's the most coherent, well thought out language I've used and I can't recommend it enough.

There is the issue of startup time with the JVM, but you can also do AOT compilation now so that really isn't a problem. Here are some other cool projects to look at if you're interested:

Malli: https://github.com/metosin/malli

Babashka: https://github.com/babashka/babashka

Clerk: https://github.com/nextjournal/clerk

BREAKING: walking a :schema passes children instead of [id] to the walker function #884

When you get to larger, more complex validations, I'd recommend checking out Malli or Spec.

for specs, you can try malli - feels pretty well supported and full featured: https://github.com/metosin/malli (i'm not 100% sure how popular it is for others, but I use it on my personal projects)

Have you tried malli: Data-driven Schemas for Clojure/Script?

All that being said, I particularly use malli and I don't find anything to complain about. There is a very nice and sound ecosystem being built around it (malli-ts is one of my contributions to it, but still in early development stages). I highly recommend reading its README, very informative stuff.

Any large codebase can be broken up into small isolated components that can be reasoned about independently. This is how you structure Clojure projects if you want them to be maintainable. Clojure inherently encourages doing this by defaulting to immutability. The contract between components is the data being passed to the component and returned by it. Using Malli schemas at the edges of the components is a typical approach to documenting their APIs https://github.com/metosin/malli

I see the fact that people often end up creating large and tightly coupled monolithic codebases in static languages as a negative aspect of static typing. Such codebases are difficult to reason about even if you have guarantees that the types align. Ultimately, you need to understand the relationships in code, and how they relate to business logic. The more coupling an application has the harder it becomes to reason about it as a whole.

Ideally, I think applications should be structured as a bunch of Lego blocks that can be composed together. Each component should encapsulate some functionality, and then the flow of the business logic should bubble up to the top and expressed in how these components are chained together.

Uhhh spec has existed for a long time and before that, schema Nowadays we also have the excellent malli. If his codebase is full of functions where the shape of the data isn’t obvious, isn’t documented and isn’t specified in a specific/schema, that’s on him and his bad coding practices and really no different from passing data in other dynamic languages. A class by itself (without additional effort) only gives you field names.