The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →
Top 23 duckdb Open-Source Projects
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
lance
Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..
-
-
rill
Rill is a tool for effortlessly transforming data sets into powerful, opinionated dashboards using SQL. BI-as-code. (by rilldata)
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
splink
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
-
-
-
-
WhatTheDuck
WhatTheDuck is an open-source web application built on DuckDB. It allows users to upload CSV files, store them in tables, and perform SQL queries on the data.
-
-
-
-
-
-
portable-data-stack-dagster
A portable Datamart and Business Intelligence suite built with Docker, Dagster, dbt, DuckDB, PostgreSQL and Superset
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Recommend checking out https://github.com/tobymao/sqlglot if you are interested in this capability for other SQL dialects
Tools like this are helpful for:
- Rendering SQL in a consistent way, eg for snapshot testing
Project mention: Show HN: Hashquery, a Python library for defining reusable analysis | news.ycombinator.com | 2024-04-23I really don't understand the appeal of dbt vs a proper programming language. The templating approach leads to massive spaghetti. I look forward to trying out something like Ibis [0]
0: https://ibis-project.org/
you should look at lance(https://lancedb.github.io/lance/)
Project mention: Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL | news.ycombinator.com | 2024-03-15Very impressive project and vision! Love the demo!
I am also ex-GS and worked on what I am fairly sure is the table display tool you're describing. I tried to carry the essential aspects of that work (multi-level pivots, with drill-down to the leaf level, and all interactive events and analytics supported by db queries) to Tad (https://www.tadviewer.com/, https://github.com/antonycourtney/tad), another open source project powered by DuckDb.
An embeddable version of Tad, powered by DuckDb WASM, is used as the results viewer in the MotherDuck Web UI (https://app.motherduck.com/).
If you're interested in embedding Tad in Pretzel, or leveraging pieces of it in your work, or collaborating on other aspects of DuckDb WASM powered UIs, please get in touch!
Project mention: Using Google Sheets as the back end/APIs of your app | news.ycombinator.com | 2024-04-12
Project mention: Splink: Fast, accurate, scalable probabilistic data linkage | news.ycombinator.com | 2024-03-13
Project mention: Show HN: JupySQL – a SQL client for Jupyter (ipython-SQL successor) | news.ycombinator.com | 2023-12-06Hey, HN community!
We're stoked to launch JupySQL today! JupySQL is an open-source library that brings a modern SQL experience to Jupyter. JupySQL is compatible with all major databases, such as Snowflake, Redshift, PostgreSQL, MySQL, MariaDB, DuckDB, SQL Server, Clickhouse, Trino, and more!
To get started, check out our tutorial: https://jupysql.ploomber.io/en/latest/quick-start.html
SQL is the defacto language for data analysis; however, analysis often requires a mix of SQL and Python. JupySQL bridges this gap, allowing users to execute SQL queries seamlessly in Jupyter and continue their analysis in Python. Add %%sql to the top of your cell and start writing SQL.
Here are some of JupySQL's main features:
- Syntax highlighting
Project mention: Shout out to Appsmith developers to check out this new tool! | /r/lowcode | 2023-07-09I am one of the members of an open-source project VulcanSQL, a Data API Framework for data applications that helps data folks create and share data APIs faster.
Project mention: Show HN: WhatTheDuck – open-source, in-browser SQL on CSV files | news.ycombinator.com | 2024-03-26
[3] https://github.com/sutoiku/puffin
One possible thing to look into would be whether this dataset is partitioned too much. My understanding is that the recommended file size for individual parquet files is 512MB to 1GB, whereas here they are 50MB. It would be interesting to see the impact of the partitioning strategy on these benchmarks.
[4] https://parquet.apache.org/docs/file-format/configurations/
0.8.0 version of DuckDB provider for .NET was released too.
Project mention: WrenAI: Open-source AI data assistant on your database. Chat With Your Database | news.ycombinator.com | 2024-04-10
Project mention: quack-reduce: duckdb as a stateless query engine over a data lake | news.ycombinator.com | 2024-01-27
Project mention: Show HN: Snowflake Data Quality Checks in Python | news.ycombinator.com | 2024-02-11
Project mention: PostGEESE? Introducing the DuckDB Spatial Extension | news.ycombinator.com | 2023-07-06
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
duckdb related posts
- Using Google Sheets as the back end/APIs of your app
- Show HN: WhatTheDuck – open-source, in-browser SQL on CSV files
- Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL
- quack-reduce: duckdb as a stateless query engine over a data lake
- JupySQL: Connecting to a SQL database from Jupyter
- GitHub - ploomber/jupysql: Better SQL in Jupyter. 📊
- SQL CTE's in Jupyter notebooks, DuckDB integration and more
-
A note from our sponsor - WorkOS
workos.com | 23 Apr 2024
Index
What are some of the best open-source duckdb projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | sqlglot | 5,441 |
| 2 | ibis | 4,074 |
| 3 | lance | 3,232 |
| 4 | tad | 3,013 |
| 5 | ingestr | 2,308 |
| 6 | rill | 1,338 |
| 7 | pretzelai | 1,331 |
| 8 | splink | 1,076 |
| 9 | dbt-duckdb | 719 |
| 10 | jupysql | 598 |
| 11 | vulcan-sql | 592 |
| 12 | inline-sql | 412 |
| 13 | WhatTheDuck | 396 |
| 14 | duckdb-rs | 357 |
| 15 | puffin | 277 |
| 16 | DuckDB.NET | 268 |
| 17 | duckdb_fdw | 230 |
| 18 | sqlite_scanner | 184 |
| 19 | WrenAI | 138 |
| 20 | quack-reduce | 116 |
| 21 | cuallee | 105 |
| 22 | portable-data-stack-dagster | 105 |
| 23 | h3-duckdb | 99 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com