Let's build a distributed Postgres proof of concept

• rqlite

  112 14,835 9.9 Go

  The lightweight, distributed relational database built on SQLite.

  

Every node in the cluster "knows" the network address of the Leader. It knows because a) it's part of the Raft cluster configuration, and every follower "heartbeats" to the Leader periodically.

So it's actually pretty simple. If a node receives a request which must be performed on the leader, and that node is not itself the Leader, it can do one of the following things:

1) reject the request with an error, but this isn't really a production-viable option.

2) reject the request with an error, but tell the client where the leader can be found, so the leader can retry the request.

3) transparently forward the request to the leader, wait for the leader to execute the request, get the respond, and return the response to the client. In this case the client doesn't even know the forwarding to the leader happened.

rqlite supports mode 2 and 3, client can choose which behavior it wants, on a request-by-request basis. Option 3 is the default.

https://github.com/rqlite/rqlite/blob/master/DOC/DATA_API.md...

• yugabyte-db

  87 8,471 10.0 C

  YugabyteDB - the cloud native distributed SQL database for mission-critical applications.

  

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• pgproto3

  1 145 0.0 Go

I was half-thinking about taking the Postgres wire protocol code[1] that CockroachDB uses, and adding it to rqlite[2]. Then you could talk to rqlite using Postgres tooling (perhaps). The only thing is that rqlite doesn't support distributed transactions in the usual sense, so there might be some difficulty mapping Postgres transaction control to rqlite. But still, could be interesting.

[1] https://github.com/jackc/pgproto3

• litestream-read-replica-example

  1 52 2.6 Go

  Discontinued An example of using Litestream's live read replication feature.

It doesn’t redirect writes but you can create read-only replicas: https://github.com/benbjohnson/litestream-read-replica-examp...

• litestream-read-replica-examp

  1 - -

It doesn’t redirect writes but you can create read-only replicas: https://github.com/benbjohnson/litestream-read-replica-examp...

• hraftd

  3 779 2.8 Go

  Discontinued A reference use of Hashicorp's Raft implementation

Yes, embedding any kind of store can work. For example, here is an in-memory KV store embedded in Raft:

https://github.com/otoolep/hraftd

This is kind of a reference use of Hashicorp's Raft.

• P

  8 2,904 8.3 C#

  The P programming language.

  

It is tough.

My approach when learning new protocols like Raft or Paxos is to implement them in Pluscal (TLA+'s higher-level language) or P (https://github.com/p-org/P). I've found that helps separate the protocol-level concerns from the implementation-level concerns (sockets? wire format?) in a way that reduces the difficulty of learning the protocol.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• orioledb

  25 2,603 9.3 C

  OrioleDB – building a modern cloud-native storage engine (... and solving some PostgreSQL wicked problems)  🇺🇦

  

OrioleDB seems like a more complete distributed postgres (https://github.com/orioledb/orioledb). It uses RAFT as well as a bunch of other changes too

• neon

  122 11,987 9.9 Rust

  Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, branching, and bottomless storage.

  

> Another approach could be using PostgreSQL as a stateless scalable query execution engine, while stubbing out the storage layer with a remote replicated consistent storage solution.

Also neon:

https://github.com/neondatabase/neon

Suggest a related project