A collection of CRDT benchmarks (by dmonad)

Crdt-benchmarks Alternatives

Similar projects and alternatives to crdt-benchmarks

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better crdt-benchmarks alternative or higher similarity.

crdt-benchmarks reviews and mentions

Posts with mentions or reviews of crdt-benchmarks. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-05-22.
  • JSON-joy CRDT benchmarks, 100x speed improvement over state-of-the-art
    4 projects | | 22 May 2023
    Author of Yjs here. I'm all for faster data structures. But only benchmarking one dimension looks quite fishy to me. A CRDT needs to be adequate at multiple dimensions. At least you should describe the tradeoffs in your article.

    The time to insert characters is the least interesting property of a CRDT. It doesn't matter to the user whether a character is inserted within .1ms or .000000001ms. No human can type that fast.

    It would be much more interesting to benchmark the time it takes to load a document containing X operations. Yjs & Yrs are pretty performant and conservative on memory here because they don't have to build an index (it's a tradeoff that we took consciously).

    When benchmarking it is important to measure the right things and interpret the results somehow so that you can give recommendations when to use your algorithm / implementation. Some things can't be fast/low enough (e.g. time to load a document, time to apply updates, memory consumption, ..) other things only need to be adequate (e.g. time to insert a character into a document).

    Unfortunately, a lot of academic papers set a bad trend of only measuring one dimension. Yeah, it's really easy to succeed in one dimension (e.g. memory or insertion-time) and it is very nice click-bait. But that doesn't make your CRDT a viable option in practice.

    I maintain a set of benchmarks that tests multiple dimensions [1]. I'd love to receive a PR from you.


  • CRDT-richtext: Rust implementation of Peritext and Fugue
    17 projects | | 18 May 2023
    Diamond types author here! Congratulations on getting your crdt working! It’s lovely to see a new generation of CRDTs which have decent performance.

    And nice stuff implementing peritext! I’d love to do the same in diamond types at some point. You beat me to it!

    Im building a little repository of real world collaborative editing traces to use when benchmarking, comparing and optimising text based CRDTs[1]. The automerge-perf editing trace isn’t enough on its own. And we’re increasingly converging on a format for multi user concurrent editing traces too[2]. It’d be great to add some rich text editing traces in the mix if you’re interested in recording something, so we can also compare how peritext performs in different systems.

    Anyway, welcome to the community! Love to have more implementations around!

  • Cloudant/IBM back off from FoundationDB based CouchDB rewrite
    3 projects | | 12 Mar 2022
    So yes, a particularly large document is not the norm but it can happen.

    JavaScript CRDTs can be quite performant, see the Yjs benchmarks:

  • Automerge: A JSON-like data structure (a CRDT) that can be modified concurrently
    12 projects | | 20 Feb 2022
  • Automerge: a new foundation for collaboration software [video]
    13 projects | | 10 Dec 2021
  • Show HN: SyncedStore CRDT – build multiplayer collaborative apps for React / Vue
    11 projects | | 8 Dec 2021
  • 5000x Faster CRDTs: An Adventure in Optimization
    8 projects | | 31 Jul 2021
    Cool! It'd be interesting to see those CRDT implementations added to Kevin Jahns' CRDT Benchmarks page[1]. The LogootSplit paper looks interesting. It looks like xray is abandoned, and I'm not sure about teletype. Though teletype's CRDT looks to be entirely implemented in javascript[2]? If the authors are around I'd love to see some benchmarks so we can compare approaches and learn what actually works well.

    And I'm not surprised these techniques have been invented before. Realising a tree is an appropriate data structure here is a pretty obvious step if you have a mind for data structures.

    To name it, I often find myself feeling defensive when people read my work and respond with a bunch of links to academic papers. Its probably totally unfair and a complete projection from my side, but I hear a voice in my head reword your comment to instead say something awful like: "Cool, but everything you did was done before. Even if they didn't make any of their work practical, usable or good they still published first and you obviously didn't do a good enough literature review if you didn't know that." And I feel an unfair defensiveness arise in me as a result that wants to find excuses to dismiss the work, even if the work might be otherwise interesting.

    Its hard to compare their benchmark results because they used synthetic randomized editing traces, which always have different performance profiles than real edits for this stuff. Their own university gathered some great real world data in an earlier study. It would have been much more instructive if that data set was used here. At a glance their RAM usage looks to be about 2 orders of magnitude worse than diamond-types or yjs. And their CPU usage... ?? I can't tell because they have no tables of results. Just some hard to read charts with log scales, so you can't even really eyeball the figures. So its really hard to tell if their work ends up performance-competitive without spending a couple days getting their enterprise style java code running with a better data set. Do you think thats worth doing?



    8 projects | | 31 Jul 2021
  • A note from our sponsor - WorkOS | 15 Apr 2024
    The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →


Basic crdt-benchmarks repo stats
2 months ago
The modern identity platform for B2B SaaS
The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.