distributed-counters
noms
distributed-counters | noms | |
---|---|---|
1 | 11 | |
6 | 7,502 | |
- | - | |
10.0 | 1.9 | |
almost 11 years ago | over 2 years ago | |
Erlang | Go | |
- | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
distributed-counters
-
Downsides of Offline First
I was also a "true believer" in CRDTs for a long time, implementing my first ones in Erlang about 9 years ago[1], but my opinion of where they fit has changed significantly.
The one issue with CRDT that I find is rarely mentioned and often ignored is the case where you've deployed these data structures that include merge logic to a set of participating nodes that you can't necessarily update at will. Think phones that people don't update, or IOT/sensor devices like electric meters or other devices "in the wild".
When you include merge logic – really any code or rules that dictate what happens when the the data of 2 or more CRDTs are merged – and you have bugs in this code running on devices you can never update, this can be a huge mess. Sure you can implement simple counters easily (like the ones I linked to), and you can even use model checking to validate them. But what about complex tree logic like for edits made to a document? Conflict resolution logic? Distributed file system operations? These are already very complex and hard to get right without multiple versions involved and unfixable bugs causing mayhem.
Having to deal with these bugs in the context of a fleet of participants on a wide range of versions of the code, the combinatorial explosion of the number of possible interactions and effects of these differing versions and bugs taken together can really become impossible to manage.
I'd be interested to hear from folks who have experience with these kinds of issues and how they have dealt with them, especially if they are still convinced that CRDTs were the right choice.
[1] https://github.com/nicolasff/distributed-counters
noms
-
How Dolt Stores Table Data
This is from 2022. It is based on Noms [1], which is no longer maintained (they forked it).
I think the Noms doc linked from this article [2] is clearer than the article itself. That said I sill cannot turn my head around to grasp how this entire thing work tbh. I hope they wrote a peer reviewed paper to serve the audience better.
[1] https://github.com/attic-labs/
[2] https://github.com/attic-labs/noms/blob/master/doc/intro.md#...
-
I was wrong. CRDTs are the future
I am. But i know very little about CRDTs lol, so we'll see how that goes. I'm interested in converting some immutable, local-first data warehouse tooling i enjoy to a CRDT version. Prior it was more.. Git-like. Basically just Git with data structures inspired-massively from Noms[1].
The thing i've found most interesting is it appears[2] that CRDT backends need to expose CRDT flavored types to users. Which is to say how i'm writing this combines the notion of a type, say `[i32]` with how you want the merges to work. CRDT works great but based on my amateur-hour researching on the subject i don't feel you can write a single CRDT merge strategy for a single data type ala `[i32]` and have it be always correct. Applications need to indicate enough context on what makes sense for a given data type.
So yea, i agree with you. I'm interested in making a database-like thing, backed by CRDTs, but i also have seen very few general purpose implementations with CRDTs. It feels like i'm breaking "new ground", while having no idea what i'm doing and having no intention of being an actual researcher here. I'm just making apps i enjoy heh.
[1]: https://github.com/attic-labs/noms
- Building a decentralized database
-
Picking low-hanging memory usage bugs of an open source database
Most of the changes are in the noms package which used to live in a separate repo (https://github.com/attic-labs/noms), but Dolt has since adopted them.
-
Downsides of Offline First
Not much more to say other than Noms was my favorite project (https://github.com/attic-labs/noms) for a while until acquisition and the engineers are now the ones behind Replicache (https://replicache.dev/).
I think this is going to be the next "Realm" that works everywhere.
- calling Format() on a time struct in a golang program changes the default Location's timezone information in the rest of the program
-
Steps to build Database System from sratch?
The storage layer based on Noms: https://github.com/attic-labs/noms
- Noms: The versioned, forkable, syncable database
-
Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
Noms might be what you’re looking for (https://github.com/attic-labs/noms). Dolt is actually a fork of Noms.
-
CondensationDB: Build secure and collaborative apps [open-source]
People that are interested in a similar feature set should check out https://github.com/attic-labs/noms and the SQL fork of Noms, https://github.com/dolthub/dolt
What are some alternatives?
offix - GraphQL Offline Client and Server
rqlite - The lightweight, distributed relational database built on SQLite.
absurd-sql - sqlite3 in ur indexeddb (hopefully a better backend soon)
dat - Go Postgres Data Access Toolkit
shelf
dolt - Dolt – Git for Data
sql-migrate - SQL schema migration tool for Go.
skeema - Declarative pure-SQL schema management for MySQL and MariaDB
cockroach - CockroachDB - the open source, cloud-native distributed SQL database.
levigo - levigo is a Go wrapper for LevelDB
ObjectBox Go Database - Embedded Go Database, the fast alternative to SQLite, gorm, etc.
FlockDB - A distributed, fault-tolerant graph database