Go distributed-database

Open-source Go projects categorized as distributed-database | Edit details

Top 6 Go distributed-database Projects

  • GitHub repo etcd

    Distributed reliable key-value store for the most critical data of a distributed system

    Project mention: etcd | reddit.com/r/JavaOnTheEdge | 2021-11-07
  • GitHub repo tidb

    TiDB is an open source distributed HTAP database compatible with the MySQL protocol

    Project mention: Comparing Nginx Performance in Bare Metal and Virtual Environments | news.ycombinator.com | 2021-10-29

    I do agree with you in that regard, however, that's also a dangerous line of thinking.

    There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility

    There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up.

    Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...

    To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal with the complexities of balancing the load and dealing with backpressure - introduce a message queue if you must, maybe even an in-memory one for simplicity's sake, or consider an event based architecture where "what needs to be done" is encapsulated within a data structure that can be passed around and applied whenever possible. In my eyes, such solutions can in many cases be better than losing the many benefits of having a single source of truth.

    Alternatively, consider sharding as a possibility, or, alternatively, do some domain driven design, figure out where to draw some boundaries and split your service into multiple ones that cover the domain with which you need to work with. Then you have one DB for sales, one for account management, one for reports and so on, all separated by something as simple as REST interfaces and with rate limits or any of the other mechanisms.

    If, however, neither of those two groups of approaches don't seem to be suitable for the loads that you're dealing with, then you probably have a team of very smart people and a large amount of resources to figure out what will work best.

    To sum up, if there are no good solutions in the space, perhaps that's because the problems themselves haven't been solved yet. Thus, sooner or later, they'll need to be sidestepped and their impact mitigated in whatever capacity is possible.

  • Nanos

    Run Linux Software Faster and Safer than Linux with Unikernels.

  • GitHub repo cockroach

    CockroachDB - the open source, cloud-native distributed SQL database.

    Project mention: Composing generic data structures in go | dev.to | 2021-11-30

    Recently a colleague, Nathan, reflecting on CockroachDB, remarked (paraphrased from memory) that the key data structure is the interval btree. The story of Nathan’s addition of the first interval btree to cockroach and the power of copy-on-write data structures is worthy of its own blog post for another day. It’s Nathan’s hand-specialization of that data structure that provided the basis (and tests) for the generalization I’ll be presenting here. The reason for this specialization was as much for the performance wins of avoiding excessive allocations, pointer chasing, and cost of type assertions when using interface boxing.

  • GitHub repo rqlite

    The lightweight, distributed relational database built on SQLite

    Project mention: Cloudflare Durable Objects Are Now Generally Available | news.ycombinator.com | 2021-11-15
  • GitHub repo Olric

    Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

    Project mention: go-generics-cache: An in-memory key:value store/cache library for Go Generics | reddit.com/r/golang | 2021-11-16

    Just to follow up -- Olric ends up being closer to what I need.

  • GitHub repo etcd-cloud-operator

    Deploying and managing production-grade etcd clusters on cloud providers: failure recovery, disaster recovery, backups and resizing.

    Project mention: Auto etcd recovery from backups? | reddit.com/r/kubernetes | 2021-09-10

    That's a thing this does! https://github.com/Quentin-M/etcd-cloud-operator

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-11-30.

Go distributed-database related posts


What are some of the best open-source distributed-database projects in Go? This list will help you:

Project Stars
1 etcd 38,050
2 tidb 29,718
3 cockroach 22,567
4 rqlite 9,085
5 Olric 2,014
6 etcd-cloud-operator 165
Find remote jobs at our new job board 99remotejobs.com. There are 33 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives