go-ds-crdt
bft-crdts
go-ds-crdt | bft-crdts | |
---|---|---|
7 | 1 | |
363 | 59 | |
2.5% | - | |
6.1 | 10.0 | |
3 months ago | over 3 years ago | |
Go | Rust | |
GNU General Public License v3.0 or later | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
go-ds-crdt
-
CRDTs Turned Inside Out
I forgot: key-value store using MD-CRDTs was implemented here: https://github.com/ipfs/go-ds-crdt
The trickiest part was not the CRDT, but the DAG traversal with multiple workers processing parallel updates on multiple branches and switching CRDT-DAG roots as they finish branches.
-
We Put IPFS in Brave
In https://github.com/ipfs/go-ds-crdt, every node in the Merkle DAG has a "Priority" field. When adding a new head, this is set to (maximum of the priorities of the children)+1.
Thus, this priority represents the current depth (or height) of the DAG at each node. It is sort of a timestamp and you could use a timestamp, or whatever helps you sort. In the case of concurrent writes, the write with highest priority wins. If we have concurrent writes of same priority, then things are sorted by CID.
The idea here is that in general, a node that is lagging behind or not syncing would have a dag with less depth, therefore its writes would have less priority when they conflict with writes from others that have built deeper DAGs. But this is after all an implementation choice, and the fact that a DAG is deeper does not mean that the last write on a key happened "later".
-
Making CRDTs Byzantine Fault Tolerant [pdf]
The idea of DAG-embedded CRDTs is far from new and was introduced here:
https://arxiv.org/abs/2004.00107 (I'm among the authors)
Unfortunately, the verification that the author proposes (not accepting new updates until the dag below is verified) will need a lot of caveats for real world usage.
Currently we use these CRDTs for a key value database of 40M+ keys in a deployment of ipfs-cluster, which uses https://github.com/ipfs/go-ds-crdt .
- Ask HN: P2P Databases?
- Go-ds-CRDT: distributed datastore using Merkle-CRDTs
- Conflict-free replicated datatypes solve distributed data consistency challenges
-
Data Laced with History: Causal Trees and Operational CRDTs (2018)
Not 100% the thing, but potentially related work in this area:
https://github.com/ipfs/go-ds-crdt
(See link to paper, and links to other projects in it, like OrbitDB).
bft-crdts
-
Making CRDTs Byzantine Fault Tolerant [pdf]
Really nice to see BFT starting to be taken seriously in CRDT research. I had done some research in this area last year and came to a lot of the same solutions (i.e. BRB protected CRDTs when dealing with VClock based CRDTs):
https://github.com/davidrusu/bft-crdts
We ended up moving away from VClock crdts entirely for our work and going with grow-only hash-graph based CRDTs as they have don't need the BRB overhead.
What are some alternatives?
merkle-crdt - Merkle-Clock CRDT implementation in python
yjs - Shared data types for building collaborative software
differential-dataflow - An implementation of differential dataflow using timely dataflow on Rust.
verneuil - Verneuil is a VFS extension for SQLite that asynchronously replicates databases to S3-compatible blob stores.
Apache Ignite - Apache Ignite
yata - YATA based algorithm for plain text CRDT edit merging in python
crdt-study - A Python study of distributed, conflict-free Last-Writer-Wins (LWW) undirected graphs