citus
hydra
citus | hydra | |
---|---|---|
61 | 26 | |
9,860 | 2,647 | |
1.4% | 4.1% | |
9.4 | 8.5 | |
10 days ago | 9 days ago | |
C | C | |
GNU Affero General Public License v3.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
citus
- SPQR 1.3.0: a production-ready system for horizontal scaling of PostgreSQL
- Citus: PostgreSQL extension that transforms Postgres into a distributed database
-
Figma's Databases team lived to tell the scale
I see they don't mention Citus (https://github.com/citusdata/citus), which is already a fairly mature native Postgres extension. From the details given in the article, in sounds like they just reimplemented it.
I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.
-
PostgreSQL Is Enough
It is possible, if you pay for it. You can do Multi-AZ Clustered Instances in RDS, where you get the benefits of Multi-AZ failover with traffic sharing.
If you can run your own infra – at least on an EC2 level – you can do things like Citus [0] for Postgres, which is about as close to "just add database nodes" as you'll get.
[0]: https://www.citusdata.com/
-
Vitess 18
So while searching for something like this for postgres I came across citus. Any one know how that stacks up?
https://github.com/citusdata/citus
- In-Depth Guide: Citus Technical Readme
-
Revolutionizing Database Scaling with CitusDB
References: CitusDB
- Squeeze the hell out of the system you have
- Show HN: Hydra 1.0 – open-source column-oriented Postgres
- Schema-based sharding comes to PostgreSQL with Citus
hydra
-
Using ClickHouse to scale an events engine
Don't feel bad, lots of people get bitten by not reading all the way down to the bottom of their readme: https://github.com/hydradatabase/hydra/blob/v1.1.2/README.md... While Hydra may very well license their own code Apache 2, they ship the AGPLv3 columnar which to my very best IANAL understanding taints the whole stack and AGPLv3's everything all the way through https://github.com/hydradatabase/hydra/blob/v1.1.2/columnar/...
-
Moving a Billion Postgres Rows on a $100 Budget
Columnar store PostgreSQL extension exists, here are two but I think I’m missing at least another one:
https://github.com/citusdata/cstore_fdw
https://github.com/hydradatabase/hydra
You can also connect other stores using the foreign data wrappers, like parquet files stored on an object store, duckdb, clickhouse… though the joins aren’t optimised as PostgreSQL would do full scan on the external table when joining.
- Hydra (YC W22) adds upsert to columnar Postgres
- Hydra
-
Is ClickHouse Moving Away from Open Source?
New column store alternative : https://github.com/hydradatabase/hydra
HN: https://news.ycombinator.com/item?id=37571974
-
Show HN: Hydra - Open-Source Columnar Postgres
some previous discussions:
https://news.ycombinator.com/item?id=37247945
https://news.ycombinator.com/item?id=36987920
and a relevant observation is that there are actually multiple license files in the repo so the consumer should read their explicit licensing section of the readme <https://github.com/hydradatabase/hydra#license> since the GitHub sidebar is misleading
-
CDC from postgres to postgres.
Hydra DB Link to Github -> Worked well for aggregated query usecases but not for queries that build reports. Also, data insertion and updation is abyssmal on columnar dbs.
-
How Query Engines Work
There's a lot of experience about db operation and how to approach MVCC encoded in PostgreSQL that shouldn't be underestimated.
[0]: https://github.com/hydradatabase/hydra
-
Hydra: Column-Oriented Postgres
And just like last time, watch out for the misleading GitHub license detector because it's not entirely Apache as the GitHub summary claims but rather *some* is Apache and buried in the interior is some AGPL stuff: https://github.com/hydradatabase/hydra#license
What are some alternatives?
Greenplum - Greenplum Database - Massively Parallel PostgreSQL for Analytics. An open-source massively parallel data platform for analytics, machine learning and AI.
duckdb - DuckDB is an in-process SQL OLAP Database Management System
yugabyte-db - YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
citus - Distributed PostgreSQL as an extension
vitess - Vitess is a database clustering system for horizontal scaling of MySQL.
ClickHouse - ClickHouse® is a free analytics DBMS for big data
TimescaleDB - An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.
postgres - PostgreSQL in Neon
dbt-core - dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
Udacity-Data-Engineering-Projects - Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
stolon - PostgreSQL cloud native High Availability and more.
vasco - vasco: MIC & MINE statistics for Postgres