go-ds-crdt VS Apache Ignite

I forgot: key-value store using MD-CRDTs was implemented here: https://github.com/ipfs/go-ds-crdt

The trickiest part was not the CRDT, but the DAG traversal with multiple workers processing parallel updates on multiple branches and switching CRDT-DAG roots as they finish branches.

In https://github.com/ipfs/go-ds-crdt, every node in the Merkle DAG has a "Priority" field. When adding a new head, this is set to (maximum of the priorities of the children)+1.

Thus, this priority represents the current depth (or height) of the DAG at each node. It is sort of a timestamp and you could use a timestamp, or whatever helps you sort. In the case of concurrent writes, the write with highest priority wins. If we have concurrent writes of same priority, then things are sorted by CID.

The idea here is that in general, a node that is lagging behind or not syncing would have a dag with less depth, therefore its writes would have less priority when they conflict with writes from others that have built deeper DAGs. But this is after all an implementation choice, and the fact that a DAG is deeper does not mean that the last write on a key happened "later".

The idea of DAG-embedded CRDTs is far from new and was introduced here:

https://arxiv.org/abs/2004.00107 (I'm among the authors)

Unfortunately, the verification that the author proposes (not accepting new updates until the dag below is verified) will need a lot of caveats for real world usage.

Currently we use these CRDTs for a key value database of 40M+ keys in a deployment of ipfs-cluster, which uses https://github.com/ipfs/go-ds-crdt .

Not 100% the thing, but potentially related work in this area:

https://github.com/ipfs/go-ds-crdt

(See link to paper, and links to other projects in it, like OrbitDB).

Ignite works as you describe:

https://ignite.apache.org/

I wouldn't really recommend this approach, I would think more in terms of subscriptions and topics and less of a 'database'.

Apache Ignite https://ignite.apache.org/

Last days, I started using Apache Ignite as a cache strategy for some applications. Apache Ignite is an open-source In-Memory Data Grid, distributed database, caching, and high-performance computing platform.