duckdb
citus
| duckdb | citus | |
|---|---|---|
| 52 | 61 | |
| 16,749 | 9,840 | |
| 4.5% | 1.2% | |
| 10.0 | 9.4 | |
| 7 days ago | 11 days ago | |
| C++ | C | |
| MIT License | GNU Affero General Public License v3.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
duckdb
- 🪄 DuckDB sql hack : get things SORTED w/ constraint CHECK
- DuckDB: Move to push-based execution model (2021)
-
DuckDB performance improvements with the latest release
I'm not sure if the fix is reassuring or not: https://github.com/duckdb/duckdb/pull/9411/files
-
Building a Distributed Data Warehouse Without Data Lakes
It's an interesting question!
The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today, the idea is that you HAVE to put it into a central location. With tools like Bacalhau[1] and DuckDB [2], you no longer have to - a single query can be sharded amongst all your data - EFFECTIVELY giving you a lot of what you want from a data lake.
It's not a replacement, but if you can do a few of these items WITHOUT moving the data, you will be able to see really significant cost and time savings.
[1] https://github.com/bacalhau-project/bacalhau
[2] https://github.com/duckdb/duckdb
- DuckDB 0.9.0
-
Push or Pull, is this a question?
[4] Switch to Push-Based Execution Model by Mytherin · Pull Request #2393 · duckdb/duckdb (github.com)
-
Show HN: Hydra 1.0 – open-source column-oriented Postgres
it depends on your query obviously.
In general, I did very deep benchmarking of pg, clickhouse and duckdb, and I sure didn't make stupid mistakes like this: https://news.ycombinator.com/item?id=36990831
My dataset has 50B rows and 2tb of data, and I think columnar dbs are very overhiped and I chose pg because:
- pg performance is acceptable, maybe 2-3x times slower than clickhouse and duckdb on some queries if pg is configured correctly and run on compressed storage
- clickhouse and duckdb start falling apart very fast because they specialized on very narrow type of queries: https://github.com/ClickHouse/ClickHouse/issues/47520 https://github.com/ClickHouse/ClickHouse/issues/47521 https://github.com/duckdb/duckdb/discussions/6696
-
🦆 Effortless Data Quality w/duckdb on GitHub ♾️
This action installs duckdb with the version provided in input.
-
Using SQL inside Python pipelines with Duckdb, Glaredb (and others?)
Duckdb: https://github.com/duckdb/duckdb - seems pretty popular, been keeping an eye on this for close to a year now.
-
CSV or Parquet File Format
The Parquet-Go library is very complex, not yet success to use it. So I ask whether DuckDB can provide API https://github.com/duckdb/duckdb/issues/7776
citus
- SPQR 1.3.0: a production-ready system for horizontal scaling of PostgreSQL
- Citus: PostgreSQL extension that transforms Postgres into a distributed database
-
Figma's Databases team lived to tell the scale
I see they don't mention Citus (https://github.com/citusdata/citus), which is already a fairly mature native Postgres extension. From the details given in the article, in sounds like they just reimplemented it.
I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.
-
PostgreSQL Is Enough
It is possible, if you pay for it. You can do Multi-AZ Clustered Instances in RDS, where you get the benefits of Multi-AZ failover with traffic sharing.
If you can run your own infra – at least on an EC2 level – you can do things like Citus [0] for Postgres, which is about as close to "just add database nodes" as you'll get.
[0]: https://www.citusdata.com/
-
Vitess 18
So while searching for something like this for postgres I came across citus. Any one know how that stacks up?
https://github.com/citusdata/citus
- In-Depth Guide: Citus Technical Readme
-
Revolutionizing Database Scaling with CitusDB
References: CitusDB
- Squeeze the hell out of the system you have
- Show HN: Hydra 1.0 – open-source column-oriented Postgres
- Schema-based sharding comes to PostgreSQL with Citus
What are some alternatives?
ClickHouse - ClickHouse® is a free analytics DBMS for big data
Greenplum - Greenplum Database - Massively Parallel PostgreSQL for Analytics. An open-source massively parallel data platform for analytics, machine learning and AI.
sqlite-worker - A simple, and persistent, SQLite database for Web and Workers.
yugabyte-db - YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
datasette - An open source multi-tool for exploring and publishing data
vitess - Vitess is a database clustering system for horizontal scaling of MySQL.
octosql - OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
TimescaleDB - An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.
metabase-clickhouse-driver - ClickHouse database driver for the Metabase business intelligence front-end
dbt-core - dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
datafusion - Apache DataFusion SQL Query Engine
stolon - PostgreSQL cloud native High Availability and more.