citus
rqlite
Our great sponsors
citus | rqlite | |
---|---|---|
61 | 112 | |
9,840 | 14,862 | |
3.6% | 1.3% | |
9.4 | 9.9 | |
5 days ago | 5 days ago | |
C | Go | |
GNU Affero General Public License v3.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
citus
- SPQR 1.3.0: a production-ready system for horizontal scaling of PostgreSQL
- Citus: PostgreSQL extension that transforms Postgres into a distributed database
-
Figma's Databases team lived to tell the scale
I see they don't mention Citus (https://github.com/citusdata/citus), which is already a fairly mature native Postgres extension. From the details given in the article, in sounds like they just reimplemented it.
I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.
-
PostgreSQL Is Enough
It is possible, if you pay for it. You can do Multi-AZ Clustered Instances in RDS, where you get the benefits of Multi-AZ failover with traffic sharing.
If you can run your own infra – at least on an EC2 level – you can do things like Citus [0] for Postgres, which is about as close to "just add database nodes" as you'll get.
[0]: https://www.citusdata.com/
-
Vitess 18
So while searching for something like this for postgres I came across citus. Any one know how that stacks up?
https://github.com/citusdata/citus
- In-Depth Guide: Citus Technical Readme
-
Revolutionizing Database Scaling with CitusDB
References: CitusDB
- Squeeze the hell out of the system you have
- Show HN: Hydra 1.0 – open-source column-oriented Postgres
- Schema-based sharding comes to PostgreSQL with Citus
rqlite
- The lightweight, easy-to-use, distributed relational database built on SQLite
-
CursusDB – A new scalable distributed document oriented database
Seems like you could do the same with rqlite [1], since SQLite supports JSON.
[1]: https://rqlite.io
-
Rqlite 8.0
rqlite[1] creator here, happy to answer any questions about rqlite, this latest release, and how it works.
[1] https://rqlite.io
-
Adding new database engine support
I found simple distributed RQlite https://github.com/rqlite/rqlite based on raft and sqlite. How hard is to add it?
- I'm All-In on Server-Side SQLite
-
So, you want to deploy on the edge?
rqlite[1] creator here, happy to answer any questions. rqlite also supports read-only nodes, which can also help with reads at the "edge". It probably wouldn't scale to 100s of nodes, it is an option.
"rqlite supports adding read-only nodes. You can use this feature to add read scalability to the cluster if you need a high volume of reads, or want to distribute copies of the data nearer to clients – but don’t want those nodes counted towards the quorum. These types of nodes are also known as non-voting nodes."
[1] https://rqlite.io/
[2] https://rqlite.io/docs/clustering/read-only-nodes/
- LiteFS Cloud: Distributed SQLite with Managed Backups
- Show HN: Rqlite, distributed DB built on SQLite, now runs on MIPS, RISC, PowerPC
- rqlite v7.19.0: the lightweight distributed relational database built on Go, Raft, and SQLite -- now runs on MIPS, PowerPC, and RISC
- rqlite v7.18: the lightweight distributed database built on Go, Raft, and SQLite -- now with new Unified HTTP endpoint for easy reads and writes
What are some alternatives?
Greenplum - Greenplum Database - Massively Parallel PostgreSQL for Analytics. An open-source massively parallel data platform for analytics, machine learning and AI.
dqlite - Embeddable, replicated and fault-tolerant SQL engine.
yugabyte-db - YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
litestream - Streaming replication for SQLite.
vitess - Vitess is a database clustering system for horizontal scaling of MySQL.
cockroach - CockroachDB - the open source, cloud-native distributed SQL database.
TimescaleDB - An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.
bolt
dbt-core - dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
etcd - Distributed reliable key-value store for the most critical data of a distributed system [Moved to: https://github.com/etcd-io/etcd]
stolon - PostgreSQL cloud native High Availability and more.
TinyGo - Go compiler for small places. Microcontrollers, WebAssembly (WASM/WASI), and command-line tools. Based on LLVM.