libclc VS yjs

This is something I planned (2015) on sharing at some point but then years flew by and here we are .. :}

It is a cacheline sized 'container' (CLC) of machine-word length records, with one record used to store the record order and remaining bits for metadata. So you can project different kinds of container semantics, such as FIFO or LRU -- any ordered set semantics -- on this meta record. Using arrays of CLCs you can create e.g. a segmented LRU, where the overhead per item is substantially less than a conventional pointer-based datastructure, and, is naturally suited for concurrent operations (for example by assigning a range to distinct worker threads), and ops require a few or couple of lines to be touched. The LRU (or whatever) semantics in aggregate will be probabilistic, as the LRU order is deterministic per unit container only. It is very fast :)

https://github.com/alphazero/libclc/blob/master/include/libc...

https://github.com/alphazero/libclc/blob/master/include/libc...

As for the non-deterministic aspects: Since container semantics e.g. LRU order is only applied at unit level, the overall cache is ~LRU. We can strictly quantify the 'ordering error' by observing the age of items in FIFO mode as they are removed: for a deterministic container the age of the item is equal to the total capacity of the queue, for a segmented (array) composed of FIFO queues, the age will have a effectively gaussian distribution around the capacity (number of units x unit capacity). But since these containers can use as few as 9 bits per entry vs 24 or more bytes for pointer based solutions (which use linked-lists), for the same allocation of memory, the capacity of the array of CLCs will be much greater, so, the distribution tail of 'short-lived' items will actually be longer lived than items in a strict queue for the same exact memory. Additional techniques, such as n-array hashing, and low order 'clock' bits at container level, can tighten this distribution significantly (i.e. ~LRU -> LRU) via better loading factors.

One idea is just to use fewer random bits in peerIDs. Yjs (https://docs.yjs.dev/) gets away with just 32 random bits. If you compromise and use 64 random bits, then even a very popular doc with 1 million lifetime peerIDs will have a < 10^-7 lifetime probability of collision.

I've seen it come up often in collaborative text editors.

Also see: https://github.com/yjs/yjs

You are absolutely right that XML is better for document structures.

My current theory is that Yjs [0] is the new JSON+XML. It gives you both JSON and XML types in one nested structure, all with conflict free merging via incremental updates.

Also, you note the issue with XML and overlapping inline markup. Yjs has an answer for that with its text type, you can apply attributes (for styling or anything else) via arbatary ranges. They can overlap.

Obviously I'm being a little hypabolic suggesting it will replace JSON, the beauty of JSON is is simplicity, but for many systems building on Yjs or similar CRDT based serialisation systems is the future.

https://github.com/yjs/yjs/

Note: https://github.com/yjs/yjs for collaborative "document edition, and user cursors"; has WebRTC, web socket, matrix.org backend

CCP needs to revamp proto anyway, due to recent exploits... practically, nothing really prevents 'em from using some sort of CRDT's to make the state of the sig view eventually consistent (yjs lib, if we're speaking frontendian).

Yjs framework: Because it is a CRDT implementation which provides collaborative editing and offline-first capability.

No problem. The database CRUD features are just helpers as an abstraction on top of yjs: https://docs.yjs.dev/. Eweser adds schemas in the form of typescript types to make using it simpler, more structured, and interoperability easier.