Go distributed-database

Open-source Go projects categorized as distributed-database | Edit details

Top 6 Go distributed-database Projects

  • etcd

    Distributed reliable key-value store for the most critical data of a distributed system

    Project mention: Context in struct | reddit.com/r/golang | 2022-01-21

    See https://github.com/etcd-io/etcd/issues/12124 for an example how they broke etcd, by renaming a package in thier code.

  • tidb

    TiDB is an open source distributed HTAP database compatible with the MySQL protocol

    Project mention: Comparing Nginx Performance in Bare Metal and Virtual Environments | news.ycombinator.com | 2021-10-29

    I do agree with you in that regard, however, that's also a dangerous line of thinking.

    There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility

    There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up.

    Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...

    To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal with the complexities of balancing the load and dealing with backpressure - introduce a message queue if you must, maybe even an in-memory one for simplicity's sake, or consider an event based architecture where "what needs to be done" is encapsulated within a data structure that can be passed around and applied whenever possible. In my eyes, such solutions can in many cases be better than losing the many benefits of having a single source of truth.

    Alternatively, consider sharding as a possibility, or, alternatively, do some domain driven design, figure out where to draw some boundaries and split your service into multiple ones that cover the domain with which you need to work with. Then you have one DB for sales, one for account management, one for reports and so on, all separated by something as simple as REST interfaces and with rate limits or any of the other mechanisms.

    If, however, neither of those two groups of approaches don't seem to be suitable for the loads that you're dealing with, then you probably have a team of very smart people and a large amount of resources to figure out what will work best.

    To sum up, if there are no good solutions in the space, perhaps that's because the problems themselves haven't been solved yet. Thus, sooner or later, they'll need to be sidestepped and their impact mitigated in whatever capacity is possible.

  • SonarLint

    Deliver Cleaner and Safer Code - Right in Your IDE of Choice!. SonarLint is a free and open source IDE extension that identifies and catches bugs and vulnerabilities as you code, directly in the IDE. Install from your favorite IDE marketplace today.

  • cockroach

    CockroachDB - the open source, cloud-native distributed SQL database.

    Project mention: CockroachDB & GKE: AUTH=implicit Backups | dev.to | 2022-01-19

    Let’s look at taking backups on a CockroachDB cluster running on Google Kubernetes Engine. For this discussion we are assuming that you have a functioning GKE environment and have installed CockroachDB using a statefulset template similar to the one found here.

  • rqlite

    The lightweight, distributed relational database built on SQLite

    Project mention: Consider SQLite | news.ycombinator.com | 2021-12-29

    I've always thought it interesting that there was a time when large(ish) websites were hosted using servers that would struggle to outperform a modern smart toaster or wristwatch, and yet modern web applications tend to demand a dramatic distributed architecture. I like the examples in this article showing what a single modern server can do when you're not scaling to Google's level.

    As an aside, what about distributed derivatives of sqlite, like rqlite, as a response to the criticism that sqlite requires your database server to also be your web server. Could something like rqlite also provide a way for an sqlite database to grow into a distributed cluster at a later point?


  • Olric

    Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

    Project mention: Olric | reddit.com/r/devopspro | 2021-12-11
  • etcd-cloud-operator

    Deploying and managing production-grade etcd clusters on cloud providers: failure recovery, disaster recovery, backups and resizing.

    Project mention: Auto etcd recovery from backups? | reddit.com/r/kubernetes | 2021-09-10

    That's a thing this does! https://github.com/Quentin-M/etcd-cloud-operator

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2022-01-21.

Go distributed-database related posts


What are some of the best open-source distributed-database projects in Go? This list will help you:

Project Stars
1 etcd 38,543
2 tidb 30,193
3 cockroach 23,138
4 rqlite 9,317
5 Olric 2,085
6 etcd-cloud-operator 174
Find remote jobs at our new job board 99remotejobs.com. There are 29 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
Less time debugging, more time building
Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.