Top 6 Go distributed-database Projects
Distributed reliable key-value store for the most critical data of a distributed systemProject mention: Context in struct | reddit.com/r/golang | 2022-01-21
See https://github.com/etcd-io/etcd/issues/12124 for an example how they broke etcd, by renaming a package in thier code.
TiDB is an open source distributed HTAP database compatible with the MySQL protocolProject mention: Comparing Nginx Performance in Bare Metal and Virtual Environments | news.ycombinator.com | 2021-10-29
I do agree with you in that regard, however, that's also a dangerous line of thinking.
There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility
There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up.
Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal with the complexities of balancing the load and dealing with backpressure - introduce a message queue if you must, maybe even an in-memory one for simplicity's sake, or consider an event based architecture where "what needs to be done" is encapsulated within a data structure that can be passed around and applied whenever possible. In my eyes, such solutions can in many cases be better than losing the many benefits of having a single source of truth.
Alternatively, consider sharding as a possibility, or, alternatively, do some domain driven design, figure out where to draw some boundaries and split your service into multiple ones that cover the domain with which you need to work with. Then you have one DB for sales, one for account management, one for reports and so on, all separated by something as simple as REST interfaces and with rate limits or any of the other mechanisms.
If, however, neither of those two groups of approaches don't seem to be suitable for the loads that you're dealing with, then you probably have a team of very smart people and a large amount of resources to figure out what will work best.
To sum up, if there are no good solutions in the space, perhaps that's because the problems themselves haven't been solved yet. Thus, sooner or later, they'll need to be sidestepped and their impact mitigated in whatever capacity is possible.
Deliver Cleaner and Safer Code - Right in Your IDE of Choice!. SonarLint is a free and open source IDE extension that identifies and catches bugs and vulnerabilities as you code, directly in the IDE. Install from your favorite IDE marketplace today.
CockroachDB - the open source, cloud-native distributed SQL database.Project mention: CockroachDB & GKE: AUTH=implicit Backups | dev.to | 2022-01-19
Let’s look at taking backups on a CockroachDB cluster running on Google Kubernetes Engine. For this discussion we are assuming that you have a functioning GKE environment and have installed CockroachDB using a statefulset template similar to the one found here.
The lightweight, distributed relational database built on SQLiteProject mention: Consider SQLite | news.ycombinator.com | 2021-12-29
I've always thought it interesting that there was a time when large(ish) websites were hosted using servers that would struggle to outperform a modern smart toaster or wristwatch, and yet modern web applications tend to demand a dramatic distributed architecture. I like the examples in this article showing what a single modern server can do when you're not scaling to Google's level.
As an aside, what about distributed derivatives of sqlite, like rqlite, as a response to the criticism that sqlite requires your database server to also be your web server. Could something like rqlite also provide a way for an sqlite database to grow into a distributed cluster at a later point?
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.Project mention: Olric | reddit.com/r/devopspro | 2021-12-11
Deploying and managing production-grade etcd clusters on cloud providers: failure recovery, disaster recovery, backups and resizing.Project mention: Auto etcd recovery from backups? | reddit.com/r/kubernetes | 2021-09-10
That's a thing this does! https://github.com/Quentin-M/etcd-cloud-operator
Go distributed-database related posts
CockroachDB & GKE: AUTH=implicit Backups
1 project | dev.to | 19 Jan 2022
Best "database" to deal with a large amount of simple data - Exact searches
2 projects | reddit.com/r/bigdata | 18 Dec 2021
Distributed SQLite for Rust
5 projects | news.ycombinator.com | 15 Dec 2021
7 projects | news.ycombinator.com | 5 Dec 2021
1 project | reddit.com/r/facepalm | 6 Nov 2021
Comparing Nginx Performance in Bare Metal and Virtual Environments
1 project | news.ycombinator.com | 29 Oct 2021
CockroachDB Grants and Schemas explained
1 project | dev.to | 28 Aug 2021
What are some of the best open-source distributed-database projects in Go? This list will help you:
Are you hiring? Post a new remote job listing for free.