-
Not quite the answer to your quest, but there are quite a few people looking at doing eventually consistent database syncing with CRDTs. Two interesting ones are:
https://electric-sql.com/ - SQLite and Postgres syncing
https://vlcn.io/ - SQLite syncing
I've been experimenting with Yjs and merging documents in database via SQL, for both SQLite and Postgres. And allowing you to query the contents of those documents, similar to how you can query JSON. Early days:
https://github.com/samwillis/yjs-sqlite-test
https://github.com/samwillis/yjs-pg-test
-
Puter.js
Puter.js - The Backend for AI-Generated Apps. One-shot full-stack apps with your existing AI coding tool. Puter.js gives you Auth, Storage, DB, AI & more, with up to 90% fewer AI tokens than other backend platforms.
-
-
Not quite the answer to your quest, but there are quite a few people looking at doing eventually consistent database syncing with CRDTs. Two interesting ones are:
https://electric-sql.com/ - SQLite and Postgres syncing
https://vlcn.io/ - SQLite syncing
I've been experimenting with Yjs and merging documents in database via SQL, for both SQLite and Postgres. And allowing you to query the contents of those documents, similar to how you can query JSON. Early days:
https://github.com/samwillis/yjs-sqlite-test
https://github.com/samwillis/yjs-pg-test
-
I agree that full stack support is the missing pice that's need to make use explode. But I do think the current implementations will get there.
CRDTs are fairly unique in that you need them to be exposed very close to the front of your stack in order to capture user intent, but you also need support further back in your stack for merging, replication, and querying.
We have good front end support and there are multiple exciting projects building collaboration servers for them. What I think is missing is support in database, that's what I've been experimenting with (below). If you are building an offline enabled app, having the ability to generate diffs and merge in database enables easy multi document sync.
Also most general purpose CRDTs are a combination of JSON and XML like data structures, it's useful to be able to query the structures in your database. For example if you build a notes app that supports inline tags, if useful to be able to query and index those from within the XML like structure without having to dump the whole thing out at another layer of your stack.
Yjs Postgres: https://github.com/samwillis/yjs-pg-test
Yjs SQLite: https://github.com/samwillis/yjs-sqlite-test
(These are just early experiments, I'm working on a cleaner shared implementation with support for various SQLite bindings, and better querying)
-
I agree that full stack support is the missing pice that's need to make use explode. But I do think the current implementations will get there.
CRDTs are fairly unique in that you need them to be exposed very close to the front of your stack in order to capture user intent, but you also need support further back in your stack for merging, replication, and querying.
We have good front end support and there are multiple exciting projects building collaboration servers for them. What I think is missing is support in database, that's what I've been experimenting with (below). If you are building an offline enabled app, having the ability to generate diffs and merge in database enables easy multi document sync.
Also most general purpose CRDTs are a combination of JSON and XML like data structures, it's useful to be able to query the structures in your database. For example if you build a notes app that supports inline tags, if useful to be able to query and index those from within the XML like structure without having to dump the whole thing out at another layer of your stack.
Yjs Postgres: https://github.com/samwillis/yjs-pg-test
Yjs SQLite: https://github.com/samwillis/yjs-sqlite-test
(These are just early experiments, I'm working on a cleaner shared implementation with support for various SQLite bindings, and better querying)
-
Cool, I'll check out your repos. Supabase also as Postgres CRDT (https://github.com/supabase/pg_crdt).
But, in general, I think searching in and across documents and database storage is going to a thorny problem even with existing, hyper-optimized CRDT algorithms. For example, if you just store your Yjs document as a binary blob in a Postgres column, Postgres becomes the bottleneck and all the fancy range-tree or b-tree optimizations are no longer that helpful on the server. Plus, it'd be great to have an document edit just be a tiny insert into the database that can be streamed to other clients rather than a big row update.
This all may because the focus of CRDTs seems to be rooted in a fully decentralized, P2P system but many developers want collaborative text editing in a more traditional client-server model.
-
The source code of the benchmark is available here https://github.com/zxch3n/fugue-bench
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
multiversion-concurrency-control
Discontinued Implementation of multiversion concurrency control, Raft, Left Right concurrency Hashmaps and a multi consumer multi producer Ringbuffer, concurrent and parallel load-balanced loops, parallel actors implementation in Main.java, Actor2.java and a parallel interpreter
https://github.com/samsquire/multiversion-concurrency-contro...
And I implemented a 3 way text diff with myers algorithm based on https://blog.jcoglan.com/2017/02/12/the-myers-diff-algorithm...
https://github.com/samsquire/text-diff
I implemented an eventually consistent mesh protocol that uses timestamps to provide last write wins
-
https://github.com/samsquire/multiversion-concurrency-contro...
And I implemented a 3 way text diff with myers algorithm based on https://blog.jcoglan.com/2017/02/12/the-myers-diff-algorithm...
https://github.com/samsquire/text-diff
I implemented an eventually consistent mesh protocol that uses timestamps to provide last write wins
-
text-diff
Discontinued a python implementation of diff3 and three way merge [GET https://api.github.com/repos/samsquire/text-diff: 404 - Not Found // See: https://docs.github.com/rest/repos/repos#get-a-repository]
https://github.com/samsquire/multiversion-concurrency-contro...
And I implemented a 3 way text diff with myers algorithm based on https://blog.jcoglan.com/2017/02/12/the-myers-diff-algorithm...
https://github.com/samsquire/text-diff
I implemented an eventually consistent mesh protocol that uses timestamps to provide last write wins
-
eventually-consistent-mesh
Discontinued an attempt to create an asynchronously replicated append only eventually consistent data protocol
https://github.com/samsquire/eventually-consistent-mesh
I implemented an barebones toy SQL and Graph cypher and document storage database
-
hash-db
Discontinued Experimental distributed pseudomultimodel keyvalue database (it uses python dictionaries) imitating dynamodb querying with join only SQL support, distributed joins and simple Cypher graph support and document storage [GET https://api.github.com/repos/samsquire/hash-db: 404 - Not Found // See: https://docs.github.com/rest/repos/repos#get-a-repository]
https://github.com/samsquire/hash-db
I need to combine the ideas in each of these projects into a cohesive solution.
I did some work on trying to implement the YATA algorithm, poorly.
-
We (ElectricSQL) have some work underway on a Rust/WASM port of our core client component, that’s designed to open up more language support.
This is a community contributed Dart/Flutter client https://github.com/SkillDevs/electric_dart that may also be a useful reference.
-
crdt-benchmarks
Discontinued Real world text editing traces for benchmarking CRDT and Rope data structures [Moved to: https://github.com/josephg/editing-traces] (by josephg)
Diamond types author here! Congratulations on getting your crdt working! It’s lovely to see a new generation of CRDTs which have decent performance.
And nice stuff implementing peritext! I’d love to do the same in diamond types at some point. You beat me to it!
Im building a little repository of real world collaborative editing traces to use when benchmarking, comparing and optimising text based CRDTs[1]. The automerge-perf editing trace isn’t enough on its own. And we’re increasingly converging on a format for multi user concurrent editing traces too[2]. It’d be great to add some rich text editing traces in the mix if you’re interested in recording something, so we can also compare how peritext performs in different systems.
Anyway, welcome to the community! Love to have more implementations around!
https://github.com/josephg/crdt-benchmarks
https://github.com/dmonad/crdt-benchmarks/issues/20
-
Diamond types author here! Congratulations on getting your crdt working! It’s lovely to see a new generation of CRDTs which have decent performance.
And nice stuff implementing peritext! I’d love to do the same in diamond types at some point. You beat me to it!
Im building a little repository of real world collaborative editing traces to use when benchmarking, comparing and optimising text based CRDTs[1]. The automerge-perf editing trace isn’t enough on its own. And we’re increasingly converging on a format for multi user concurrent editing traces too[2]. It’d be great to add some rich text editing traces in the mix if you’re interested in recording something, so we can also compare how peritext performs in different systems.
Anyway, welcome to the community! Love to have more implementations around!
https://github.com/josephg/crdt-benchmarks
https://github.com/dmonad/crdt-benchmarks/issues/20
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Related posts
-
Show HN: RemoteStorage – sync localStorage across devices and browsers
-
Cloudflare acquires PartyKit to allow developers to build real-time multi-user
-
Show HN: PowerSync – Bi-directional Postgres<>SQLite sync for offline-first apps
-
Electric SQL – Local-first sync layer for web and mobile apps
-
Loro Now Open Source: Reimagine State Management with CRDTs