diamond-types VS interface-types

> I think far more interesting these days would be projects like Veilid, Hyphanet's Locutus

I have not assessed Veilid yet but it's on my list and at a first glance seems like a very serious and informed attempt. I'm personal friends with Freenet / Hyphanet's Ian Clarke and spoke with him about Locutus when he was just getting started. It sounded awesome then and I will give this a second look too, though when he explained it to me it sounded like it had the same limitations with deletion that Nostr or the global IPFS network would have. It does seem important to note here that both Veilid and Locutus are much less mature and battle-tested than libp2p and Tor and have less Lindy longevity (longevity as a function of age.) We already suffer a lot from being on the bleeding edge, so it's nice to limit the number of bleeding edge tools we use. Libp2p, notably, has been rock solid for us and barely a time drain at all, apart from some unexpected interactions with Tor which are mostly about the lack of an official first-class Tor transport, which is specific to our use case and should start to change soon when Tor's Arti is ready.

> and ultimately Nostr -- even though not truly P2P in that sense -- which already happens to have a first try going with nostrchat.io.

Nostr and Bluesky both seem very promising for the open-world use case of social networking, and it has been amazing to see Nostr grow so rapidly as a community. I am rooting for this project and we might use it someday in Quiet for public feeds. Timed deletion is the user requirement that drives me away from building Quiet on Nostr. Based on conversations I've had with users doing sensitive work (and based on my own experience as a founder of Fight for the Future) timed deletion is extremely important to team security, and for deletion to be meaningful one needs more control over where the data is relayed than what Nostr provides in the default mode. A group that wanted trustworthy timed deletion would have to control their own private Nostr relay. Technically, a Tor relay could subvert the timed deletion of some Quiet messages just by capturing all traffic, but this is much less of a worry.

> If P2P is something that is truly desired, I feel like projects like Briar (https://briarproject.org/how-it-works/) have solved this with Bramble (https://code.briarproject.org/briar/briar-spec/blob/master/p...) more eloquently than it could be done on top of IPFS.

Bramble could work for us and I would recommend that anyone look into it. Briar is probably the most similar thing to Quiet that exists right now. There are big differences between Quiet and Briar, but we could definitely build Quiet on Bramble if it adequately supports iOS. My worry would be its maturity as a tool for people building things other than Briar. That could be worth the risk though and I do recommend anyone else reading this thread look at Bramble if you are doing something similar.

> I could nevertheless imagine it being overtaken fairly quickly by other projects sporting a rather lightweight and more managable basis, that allows for increased development speed and ultimately for faster iteration on features that users might wish for (e.g. DMs, @-mentions, message deletion, mobile clients, you-name-it) -- without the need to invest heavily into e.g. performance (or reliability!) issues of the underlying framework.

This is definitely something we will keep an eye on, and thank you for the thoughtful advice! My guess is that as soon as we have a significant number of real users we will need to build things that don't happen to be supported by whatever stack we choose (whether that is our current stack, Bramble, Veilid, Automerge, etc.) So the question is what's the easiest one to maintain and adapt. So far libp2p and IPFS have both been good in that department: implementations in many languages, active development, an absence of major problems showing signs of maturity (especially in libp2p), etc.

Also, my 2 cents are (for anyone following along) that if I had to do this all over again I would use Tor + Libp2p + Automerge. Libp2p and Gossipsub are solid, flexible, and will be around a while. No need to reinvent the wheel. The conceptual framework behind Automerge and Briar/Bramble are pretty similar (sync state!) but the Automerge team exists to serve people building other apps, while the Bramble team mostly focuses on Briar AFAIK. What's nice about Automerge is that the community around it (Ink & Switch, Martin Kleppmann, and other academics) is all at the academic frontier, so the level of thought and anticipation of user needs that goes into their decisions is very thorough, even if the implementations lag behind the papers. If I was doing real-time text I would also look at the Briar project and Seph Gentle's work on Diamond Types, since that's where the most thought has gone into the raw performance you need for text CRDTs that can handle large documents: https://github.com/josephg/diamond-types

But I think thats about it. Maybe there's more manually specified types in "normal" rust because most functions are smaller than that. But, it doesn't feel so bad. In this case I could probably even remove the explicit type annotation for that queue definition if I wanted to, but it makes the compiler's errors better leaving it in.

[1] https://github.com/josephg/diamond-types/blob/66025b99dbe390...

diamond-types (for reference for others [0]) still only supports plain text, is that right? I was thinking of using it for more general use cases such as an offline habit tracker, which isn't text of course, but I was interested to hear more on the progress towards other data types such as generic JSON data.

Currently for this use case I've been using autosurgeon [1] so far which has a nice Rust API for structs, even if it might be slower than yjs (or yrs, its Rust implementation) or diamond-types.

[0] https://github.com/josephg/diamond-types

[1] https://github.com/automerge/autosurgeon

I'm working on a CRDT to solve this problem too[1]. How do you plan on implementing collaborative text editing on top of your event-reordering system? Off the top of my head I can't think of a way to implement text on your proposed system which would be performant and simple.

[1] https://github.com/josephg/diamond-types

For example, this file implements a complex iterator via a struct and really complex next() method. This file was about 1/3rd the size before I manually rewrote it into a "continuation passing" style. I find it significantly harder to read and maintain in its current form.

> In this case, the bottleneck at 9 million LoC is not CPU cycles but memory usage. That's where I am considering pushing down into WebAssembly

How often does this come up in practice? I can't think of many files I've opened which were 9 million lines long. And you say "LoC" (lines of code). Are you doing syntax highlighting on 9 million lines of source code in javascript? Thats impressive!

> I guess my point is why do you need balanced trees? Is this a CRDT specific thing? Can you implement CRDT with just an array of lines / gap buffer?

Of course! Its just going to be slower. I made a simple reference implementation of Yjs, Automerge and Sync9's list types in javascript here[1]. This code is not optimized, and it takes 30 seconds to process an editing trace that diamond types (in native rust) takes 0.01 seconds to process. We could speed that up - yjs does the same thing in 1 second. But I don't think javascript will ever run as fast as optimized rust code.

The b-tree in diamond types is used for merging. If you're merging 2 branches, we need to map insert locations from the incoming branch into positions in the target (merged) branch. As items are inserted, the mapping changes dynamically. The benchmark I've been using for this is how long it takes to replay (and re-merge) all the changes in the most edited file in the nodejs git repository. That file has just shy of 1M single character insert / delete operations. If you're curious, the causal graph of changes looks like this[2].

Currently it takes 250ms to re-merge the entire causal graph. This is much slower than I'd like, but we can cache the merged positions in about 4kb on disk or something so we only need to do it once. I also want to replace the b-tree with a skip list. I think that'll make the code faster and smaller.

A gap buffer in javascript might work ok... if you're keen, I'd love to see that benchmark. The code to port is here: [3]

> Undo support -> In which case, you only have to stack / remember the set of commands and not have to store the state on every change. I'm not sure if this overlaps with the data structure choice, other than implementation details.

Yeah, I basically never store a snapshot of the state. Not on every change. Not really at all. Everything involves sending around patches. But you can't just roll back the changes when you undo.

Eg: I type "aaa" at position 0 (the start of the document). You type "bbb" at the start of the document. The document is now "bbbaaa". I hit undo. What should happen? Surely, we delete the "aaa" - now at position 3.

Translating from position 0 to position 3 is essentially the same algorithm we need to run in order to merge.

> I was just looking into TypedArrays.

I tried optimizing a physics library a few years ago by putting everything in typedarrays and it was weirdly slower than using raw javascript arrays. I have no idea why - but maybe thats fixed now.

TypedArrays are useful, but they're no panacea. You could probably write a custom b-tree on top of a typedarray in javascript if you really want to - assuming your data also fits into typedarrays. But at that point you may as well just use wasm. It'll be way faster and more ergonomic.

[1] https://github.com/josephg/reference-crdts

[2] https://home.seph.codes/public/node_graph.svg

[3] https://github.com/josephg/diamond-types/tree/master/src/lis...

Hi everyone! Author here. I'm happy to answer questions.

I wrote this a couple years ago. Since then I've been working on my own CRDT called Diamond Types[1], which uses a lot of these ideas to be bonkers fast. I've built several OT based collaborative editing systems, and diamond types is much faster than any of them - though rust and wasm might be the real MVPs here. I wrote a follow-up to this article last year when I got that working, talking about how some of the optimizations work. That article is here[2].

A fair bit has changed since I wrote that article. Yjs has started a rewrite in rust (called yrs[3]). And Automerge has apparently dramatically improved performance based on some of the ideas I talk about in this article. Oh, and diamond types has been rewritten from the ground up. Its now about 5x faster than it was last year, by completely changing the internal structure. But thats a story for another day.

Unfortunately I still only support collaborative text editing. Adding full JSON support comes soon, after I document some more of the tricks I'm doing. Its really fun work!

Why do I only support collaborative text editing? Because I care about performance, and text CRDT performance is hard because you have so many individual changes. (One for each keystroke!). Making text editing fast means everything is fast. But we've still got to do the work. To make that happen, my plan is to add full JSON editing support to diamond types using shelf[4]. Shelf is a super simple CRDT which fits in 100 lines of javascript.

[1] https://github.com/josephg/diamond-types/

[2] https://josephg.com/blog/crdts-go-brrr/

[3] https://github.com/y-crdt/y-crdt/tree/main/yrs

[4] https://github.com/dglittle/shelf

Yep. I've done something very similar on top of Diamond Types for a little personal wiki. This page[1] is synced between all users who have the page open. Its a remarkably small piece of code, outside of the CRDT library itself (which is in rust via wasm). The way it works is:

- On page load, the server sends the whole CRDT document to the browser, and the server streams changes from that point onwards.

- When a change happens in the browser, it makes that change locally then and sends anything the server doesn't know about upstream.

- Whenever the server finds out about a new change, it re-broadcasts that change to any subscribed browser streams.

I'm using the braid HTTP protocol for changes - but we could easily switch to a SSE or websocket solution. It doesn't really matter.

At the moment I'm just using flat files for storage, but there's nothing stopping you using a database instead, except that its a bit awkward to use efficient CRDT packing techniques in a database.

[1] https://wiki.seph.codes/hn

Code is here, if anyone is interested. The whole thing is a few hundred lines all up: https://github.com/josephg/diamond-types/tree/0cb5d7ecf49364...

With each change we just send the missing operations. Https://wiki.seph.codes/reddit if you want to mess around and see it in action via wasm. The code which runs this wiki is here.

Another way to trace allocations in rust is to inject some code in a global allocator. Then you can use any in-program code you like to print / track / trace allocations. For example, I wrote this code in a library I’m working on so I can track and print out how many total bytes have been allocated, and how many allocation calls have been made.

Some things that might greatly increase wasm usage and overall tooling:

1) Tools that run docker containers and serverless function services (like AWS lambda) to support providing a .wasm files instead

2) Garbage collection in the runtime to make GC languages easier to port to wasm

3) Dynamically typed languages (NodeJS, Python, Ruby) being able to compile to webassembly directly instead of porting the runtime to webassembly and then running the code through the runtime. This is a big ask though, basically needs to redesign the runtime completely

4) wasm-DOM bindings will enable other languages to do HTML rendering which will require new web frameworks for every language that wants to take over the space from JS. This will lead to (even more) fragmentation of the web ecosystem

5) A new wasm-first SDK (unrelated to the DOM) for building cross platform applications. I can see this taking off only if it is built-into the browsers and backed by some standards committee, so not very likely I think

6) Something like the Interface Types proposal ( https://github.com/WebAssembly/interface-types/blob/main/pro... ) becomes a thing allowing wasm programs to be consisted of modules written in several different languages and being able to call said modules with low or 0 runtime performance hit (and of course, no compilation to multiple CPU archs). So much of programming ecosystems are locked to specific languages (like data science with python) when there is little technical reason for it be like that.

AFAIK GC is irrelevant for "direct DOM access", you would rather want to hop into the following rabbit hole:

- reference types: https://github.com/WebAssembly/reference-types/blob/master/p...

- interface types (inactive): https://github.com/WebAssembly/interface-types/blob/main/pro...

- component model: https://github.com/WebAssembly/component-model

If this looks like a mess, that's because it is. Compared to that, the current solution to go through a Javascript shim doesn't look too bad IMHO.

I'm not really optimistic that particular aspect will get much improvement. Many people expected interface types to come save the day, but after a looong stagnation that proposal has been archived (for now) in favour of component types, which has much less potential for performance gains.

Really good questions. Unfortunately, most of the issues I found back then were fundamental ones. I've seen that Wasm has deprecated "Interface Types" and is now working on the "Component Model". But even then, as far as I understand that would only avoid the serialization and deserialization steps, and you would still need to copy complex types. It will be more performant, but I don't think it would be enough for Tremor either.

wasm doesn't know anything about the outside world on purpose. This allows it to be used in other domains. For direct access to the DOM et al, interface types are being developed. It's a non-trivial problem to interoperate with a dynamically typed GC'd language from any statically typed no-GC language that can compile to wasm.

You may want to look into WASM interface types, which is defining what amounts to am IDL for WASM and different languages have common calling conventions: https://hacks.mozilla.org/2019/08/webassembly-interface-type...

I don’t know if there’s a better intro article. I believe this is the current iteration of the proposal: https://github.com/WebAssembly/interface-types/blob/main/pro...

Yes, and if I'm not mistaken, this is the RFC

I see no reason why not. See the interface types proposal for a proposed solution.

Wanted to get thoughts from the Rust experts on this - the author of the Yew framework seems to think that Web Assembly Interface Types (https://github.com/WebAssembly/interface-types/blob/master/proposals/interface-types/Explainer.md) will allow Yew to eventually become faster than Vue, React, Angular, etc. Is there general consensus on this in the Rust community? The prospect of mixing Rust (for the performance critical pieces) with TS on the front end doesn't seem super appealing to me.