go-ds-crdt
Apache Ignite
go-ds-crdt | Apache Ignite | |
---|---|---|
7 | 3 | |
363 | 4,693 | |
2.5% | 0.6% | |
6.1 | 9.5 | |
3 months ago | 5 days ago | |
Go | Java | |
GNU General Public License v3.0 or later | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
go-ds-crdt
-
CRDTs Turned Inside Out
I forgot: key-value store using MD-CRDTs was implemented here: https://github.com/ipfs/go-ds-crdt
The trickiest part was not the CRDT, but the DAG traversal with multiple workers processing parallel updates on multiple branches and switching CRDT-DAG roots as they finish branches.
-
We Put IPFS in Brave
In https://github.com/ipfs/go-ds-crdt, every node in the Merkle DAG has a "Priority" field. When adding a new head, this is set to (maximum of the priorities of the children)+1.
Thus, this priority represents the current depth (or height) of the DAG at each node. It is sort of a timestamp and you could use a timestamp, or whatever helps you sort. In the case of concurrent writes, the write with highest priority wins. If we have concurrent writes of same priority, then things are sorted by CID.
The idea here is that in general, a node that is lagging behind or not syncing would have a dag with less depth, therefore its writes would have less priority when they conflict with writes from others that have built deeper DAGs. But this is after all an implementation choice, and the fact that a DAG is deeper does not mean that the last write on a key happened "later".
-
Making CRDTs Byzantine Fault Tolerant [pdf]
The idea of DAG-embedded CRDTs is far from new and was introduced here:
https://arxiv.org/abs/2004.00107 (I'm among the authors)
Unfortunately, the verification that the author proposes (not accepting new updates until the dag below is verified) will need a lot of caveats for real world usage.
Currently we use these CRDTs for a key value database of 40M+ keys in a deployment of ipfs-cluster, which uses https://github.com/ipfs/go-ds-crdt .
- Ask HN: P2P Databases?
- Go-ds-CRDT: distributed datastore using Merkle-CRDTs
- Conflict-free replicated datatypes solve distributed data consistency challenges
-
Data Laced with History: Causal Trees and Operational CRDTs (2018)
Not 100% the thing, but potentially related work in this area:
https://github.com/ipfs/go-ds-crdt
(See link to paper, and links to other projects in it, like OrbitDB).
Apache Ignite
-
Ask HN: P2P Databases?
Ignite works as you describe:
https://ignite.apache.org/
I wouldn't really recommend this approach, I would think more in terms of subscriptions and topics and less of a 'database'.
-
Which library/project do you wish was ported to golang?
Apache Ignite https://ignite.apache.org/
-
.NET and Apache Ignite: Testing Cache and SQL API features — Part I
Last days, I started using Apache Ignite as a cache strategy for some applications. Apache Ignite is an open-source In-Memory Data Grid, distributed database, caching, and high-performance computing platform.
What are some alternatives?
merkle-crdt - Merkle-Clock CRDT implementation in python
LiteDB - LiteDB - A .NET NoSQL Document Store in a single data file
differential-dataflow - An implementation of differential dataflow using timely dataflow on Rust.
MongoDB - The MongoDB Database
verneuil - Verneuil is a VFS extension for SQLite that asynchronously replicates databases to S3-compatible blob stores.
Alluxio (formerly Tachyon) - Alluxio, data orchestration for analytics and machine learning in the cloud
yjs - Shared data types for building collaborative software
Event Store - EventStoreDB, the event-native database. Designed for Event Sourcing, Event-Driven, and Microservices architectures
yata - YATA based algorithm for plain text CRDT edit merging in python
SqlKata Query Builder - SQL query builder, written in c#, helps you build complex queries easily, supports SqlServer, MySql, PostgreSql, Oracle, Sqlite and Firebird
crdt-study - A Python study of distributed, conflict-free Last-Writer-Wins (LWW) undirected graphs
Insight.Database - Fast, lightweight .NET micro-ORM