duckdb
tsbs
Our great sponsors
duckdb | tsbs | |
---|---|---|
51 | 76 | |
15,710 | 1,198 | |
10.4% | 2.1% | |
10.0 | 1.9 | |
4 days ago | 7 days ago | |
C++ | Go | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
duckdb
-
DuckDB performance improvements with the latest release
I'm not sure if the fix is reassuring or not: https://github.com/duckdb/duckdb/pull/9411/files
Just had a look (https://github.com/duckdb/duckdb/issues/9399). Yeah it's worrying that such a trivial query returned incorrect results - but credit to the Devs for getting it fixed quickly.
To my knowledge the only databases that can be described as "military-grade" in terms of testing are SQLite and Postgres.
-
Building a Distributed Data Warehouse Without Data Lakes
It's an interesting question!
The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today, the idea is that you HAVE to put it into a central location. With tools like Bacalhau[1] and DuckDB [2], you no longer have to - a single query can be sharded amongst all your data - EFFECTIVELY giving you a lot of what you want from a data lake.
It's not a replacement, but if you can do a few of these items WITHOUT moving the data, you will be able to see really significant cost and time savings.
- DuckDB 0.9.0
-
Push or Pull, is this a question?
[4] Switch to Push-Based Execution Model by Mytherin · Pull Request #2393 · duckdb/duckdb (github.com)
-
Show HN: Hydra 1.0 – open-source column-oriented Postgres
it depends on your query obviously.
In general, I did very deep benchmarking of pg, clickhouse and duckdb, and I sure didn't make stupid mistakes like this: https://news.ycombinator.com/item?id=36990831
My dataset has 50B rows and 2tb of data, and I think columnar dbs are very overhiped and I chose pg because:
- pg performance is acceptable, maybe 2-3x times slower than clickhouse and duckdb on some queries if pg is configured correctly and run on compressed storage
- clickhouse and duckdb start falling apart very fast because they specialized on very narrow type of queries: https://github.com/ClickHouse/ClickHouse/issues/47520 https://github.com/ClickHouse/ClickHouse/issues/47521 https://github.com/duckdb/duckdb/discussions/6696
-
🦆 Effortless Data Quality w/duckdb on GitHub ♾️
This action installs duckdb with the version provided in input.
-
Using SQL inside Python pipelines with Duckdb, Glaredb (and others?)
Duckdb: https://github.com/duckdb/duckdb - seems pretty popular, been keeping an eye on this for close to a year now.
-
CSV or Parquet File Format
The Parquet-Go library is very complex, not yet success to use it. So I ask whether DuckDB can provide API https://github.com/duckdb/duckdb/issues/7776
-
DuckDB 0.8.0
Another cool new feature that's not mentioned in the blog post is function chaining:
https://github.com/duckdb/duckdb/pull/6725
I've been using DuckDB for filtering and post-processing data, specially strings, and this will make writing complex queries easier. By combining nested functions[0] and text functions[1], sometimes I don't even need to go into a Python notebook.
tsbs
-
Fuzz Testing Is the Best Thing to Happen to Our Application Tests
1. correctness: from small units tests to relatively complex integrations tests. they typically populate a test database and query it via various interfaces, such as REST or the Postgres protocol. we use Azure Pipelines to execute them - testing in MacoOS, Linux (both Intel and ARM) and Windows.
2. performance: we tend to use the TSBS project for most of our performance testing and profiling. fun fact: we actually had to patch it as the vanilla TSBS was a bottleneck in some tests. Sadly, the PR with the improvements is still not merged: https://github.com/timescale/tsbs/pull/186
-
MongoDB Time Series Benchmark and Review
As usual, we use the industry standard Time Series Benchmark Suite (TSBS) as the benchmark tool. Unfortunately, TSBS upstream does not support MongoDB time series collections.
-
Show HN: QuestDB with Python, Pandas and SQL in a Jupyter notebook – no install
yes correct - although Clickhouse is more of an OLAP database. Timescale is built on top of Postgres, while QuestDB is built from scratch with Postgres wire compatibility. You can run benchmarks on https://github.com/timescale/tsbs
-
Streaming data storage
According their benchmark it is really fast.
-
Ingesting with CrateDB
We used the nodeIngestBench for all the benchmarking. It is a multi-process Node.js script that runs high-performance ingest benchmarks on CrateDB. It uses a data model that was adapted from Timescale’s Time Series Benchmark Suite (TSBS). One thing that we want to make clear is that nodeIngestBench is a write benchmark. The data structure that it creates is unsuitable for any performance-indicative reading tests because of its high cardinality (due to random data) and no partitioning.
-
4Bn rows/sec query benchmark: Clickhouse vs QuestDB vs Timescale
In order to make the benchmark easily reproducible, we're going to use TSBS benchmark utilities to generate the data. We'll be using so-called IoT use case:
-
DeWitt Clause, or Can You Benchmark %DATABASE% and Get Away With It
Also, some open-source vendors collaboratively maintain benchmarking suites such as Time Series Benchmark Suite to help choose the best tools for particular workloads.
-
4Bn rows/SEC query benchmark: ClickHouse vs. QuestDB vs. Timescale
Last year we released QuestDB 6.0 and achieved an ingestion rate of 1.4 million rows per second (per server). We compared those results to popular open source databases [1] and explained how we dealt with out of order ingestion under the hood while keeping the underlying storage model read-friendly. Since then, we focused our efforts on making queries faster, in particular filter queries with WHERE clauses. To do so, we once again decided to make things from scratch and built a JIT (Just-in-Time) compiler for SQL filters, with tons of low-level optimisations such as SIMD. We then parallelized the query execution to improve the execution time even further. In this blog post, we first look at some benchmarks against Clickhouse and TimescaleDB, before digging deeper in how this all works within QuestDB's storage model. Once again, we use the Time Series Benchmark Suite (TSBS) [2], developed by TimescaleDB,: it is an open source and reproducible benchmark.
We'd love to get your feedback!
This table schema: https://github.com/timescale/tsbs/blob/bcc00137d72d889e6059e...
...seems like a quite odd way to store time-series in ClickHouse. If I understood that code correctly (and I am really not sure), they partition their data by some tag value (the first one in a list?) instead of time, which is what timescaledb afaik partitions by. Of course that query filtering by timerange is going to be slower than usual. Whether that makes sense depends on your usecase.
What are some alternatives?
ClickHouse - ClickHouse® is a free analytics DBMS for big data
sqlite-worker - A simple, and persistent, SQLite database for Web and Workers.
datasette - An open source multi-tool for exploring and publishing data
octosql - OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
metabase-clickhouse-driver - ClickHouse database driver for the Metabase business intelligence front-end
QuestDB - An open source time-series database for fast ingest and SQL queries
arrow-datafusion - Apache Arrow DataFusion SQL Query Engine
LevelDB - LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
duckdb-rs - Ergonomic bindings to duckdb for Rust
TimescaleDB - An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.
toydb - Distributed SQL database in Rust, written as a learning project