go-ds-crdt
yjs
Our great sponsors
go-ds-crdt | yjs | |
---|---|---|
7 | 53 | |
357 | 15,150 | |
1.7% | 4.7% | |
6.1 | 8.4 | |
3 months ago | 3 days ago | |
Go | JavaScript | |
GNU General Public License v3.0 or later | MIT |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
go-ds-crdt
-
CRDTs Turned Inside Out
I forgot: key-value store using MD-CRDTs was implemented here: https://github.com/ipfs/go-ds-crdt
The trickiest part was not the CRDT, but the DAG traversal with multiple workers processing parallel updates on multiple branches and switching CRDT-DAG roots as they finish branches.
-
We Put IPFS in Brave
In https://github.com/ipfs/go-ds-crdt, every node in the Merkle DAG has a "Priority" field. When adding a new head, this is set to (maximum of the priorities of the children)+1.
Thus, this priority represents the current depth (or height) of the DAG at each node. It is sort of a timestamp and you could use a timestamp, or whatever helps you sort. In the case of concurrent writes, the write with highest priority wins. If we have concurrent writes of same priority, then things are sorted by CID.
The idea here is that in general, a node that is lagging behind or not syncing would have a dag with less depth, therefore its writes would have less priority when they conflict with writes from others that have built deeper DAGs. But this is after all an implementation choice, and the fact that a DAG is deeper does not mean that the last write on a key happened "later".
-
Making CRDTs Byzantine Fault Tolerant [pdf]
The idea of DAG-embedded CRDTs is far from new and was introduced here:
https://arxiv.org/abs/2004.00107 (I'm among the authors)
Unfortunately, the verification that the author proposes (not accepting new updates until the dag below is verified) will need a lot of caveats for real world usage.
Currently we use these CRDTs for a key value database of 40M+ keys in a deployment of ipfs-cluster, which uses https://github.com/ipfs/go-ds-crdt .
- Ask HN: P2P Databases?
- Go-ds-CRDT: distributed datastore using Merkle-CRDTs
- Conflict-free replicated datatypes solve distributed data consistency challenges
-
Data Laced with History: Causal Trees and Operational CRDTs (2018)
Not 100% the thing, but potentially related work in this area:
https://github.com/ipfs/go-ds-crdt
(See link to paper, and links to other projects in it, like OrbitDB).
yjs
- Show HN: Collaborate on your YC Application with CRDT-powered forms
-
Making CRDTs 98% More Efficient
One idea is just to use fewer random bits in peerIDs. Yjs (https://docs.yjs.dev/) gets away with just 32 random bits. If you compromise and use 64 random bits, then even a very popular doc with 1 million lifetime peerIDs will have a < 10^-7 lifetime probability of collision.
-
An Interactive Intro to CRDTs
I've seen it come up often in collaborative text editors.
Also see: https://github.com/yjs/yjs
-
JSON Schema Store
You are absolutely right that XML is better for document structures.
My current theory is that Yjs [0] is the new JSON+XML. It gives you both JSON and XML types in one nested structure, all with conflict free merging via incremental updates.
Also, you note the issue with XML and overlapping inline markup. Yjs has an answer for that with its text type, you can apply attributes (for styling or anything else) via arbatary ranges. They can overlap.
Obviously I'm being a little hypabolic suggesting it will replace JSON, the beauty of JSON is is simplicity, but for many systems building on Yjs or similar CRDT based serialisation systems is the future.
https://github.com/yjs/yjs/
-
Launch HN: Tiptap (YC S23) – Toolkit for developing collaborative editors
Note: https://github.com/yjs/yjs for collaborative "document edition, and user cursors"; has WebRTC, web socket, matrix.org backend
-
Wormholers, what can CCP and wormholers do to improve J-Space?
CCP needs to revamp proto anyway, due to recent exploits... practically, nothing really prevents 'em from using some sort of CRDT's to make the state of the sig view eventually consistent (yjs lib, if we're speaking frontendian).
-
How to use Yjs with Ruby on Rails?
Yjs framework: Because it is a CRDT implementation which provides collaborative editing and offline-first capability.
-
🐑🐑🐑 EweserDB, the user-owned database 🐑🐑🐑
No problem. The database CRUD features are just helpers as an abstraction on top of yjs: https://docs.yjs.dev/. Eweser adds schemas in the form of typescript types to make using it simpler, more structured, and interoperability easier.
- Ask HN: What is new in Algorithms / Data Structures these days?
- How does Google docs send the changes done by other users in real-time?
What are some alternatives?
merkle-crdt - Merkle-Clock CRDT implementation in python
automerge - A JSON-like data structure (a CRDT) that can be modified concurrently by different users, and merged again automatically.
differential-dataflow - An implementation of differential dataflow using timely dataflow on Rust.
liveblocks - Liveblocks is a platform to ship collaborative features like comments, notifications, text editors in minutes instead of months.
verneuil - Verneuil is a VFS extension for SQLite that asynchronously replicates databases to S3-compatible blob stores.
automerge-rs - Rust implementation of automerge [Moved to: https://github.com/automerge/automerge]
Apache Ignite - Apache Ignite
crdt-woot - Implementation of collaborative editing algorithm CRDT WOOT.
yata - YATA based algorithm for plain text CRDT edit merging in python
milkdown - 🍼 Plugin driven WYSIWYG markdown editor framework.
crdt-study - A Python study of distributed, conflict-free Last-Writer-Wins (LWW) undirected graphs
MobX - Simple, scalable state management.