go-duckdb
Our great sponsors
go-duckdb | postgres_scanner | |
---|---|---|
4 | 6 | |
488 | 170 | |
- | 5.9% | |
8.2 | 9.3 | |
14 days ago | 10 days ago | |
Go | C++ | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
go-duckdb
-
Embeddable Database for Go which have Date/Time type
DuckDB also has date functions and Go bindings
-
Range Joins in DuckDB
I've been beating my head trying to get duckdb to statically link into a Go program (I'm neither an expert with cgo nor ld). If anyone else has been able to do this I'd love to see your build steps.
https://github.com/marcboeker/go-duckdb produces a non-static binary by default.
-
Friendlier SQL with DuckDB
Here is a solved Github Issue related to CGO for the Go bindings! If you have another issue, please feel free to post it on their Github page!
-
Dsq: Commandline tool for running SQL queries against JSON, CSV, Parquet, etc.
Yeah frankly the q benchmark isn't the best even though dsq compares favorably in it. It isn't well documented and exercises a very limited amount of functionality and isn't very rigorous from what I can see. That said, the caching q does is likely very solid (and not something dsq does).
The biggest risk I think with octosql (and cube2222 is here somewhere to disagree with me if I'm wrong) is that they have their own entire SQL engine whereas textql, q and dsq use SQLite. But q is also in Python whereas textql, octosql, and dsq are in Go.
In the next few weeks I'll be posting some benchmarks that I hope are a little fairer (or at least well-documented and reproducible). Though of course it would be appropriate to have independent benchmarks too since I now have a dog in the fight.
On a tangent, once the go-duckdb binding [0] matures I'd love to offer duckdb as an alternative engine flag within dsq (and DataStation). Would be neat to see.
postgres_scanner
-
Connect ODBC Databases to DuckDB
I've created an ODBC DuckDB extension to query any database that has an ODBC driver. It's modeled after the fantastic official Postgres scanner extension https://github.com/duckdblabs/postgres_scanner.
It supports fetching rowsets in batches to minimize network overhead and defaults to the default DuckDB vector size of 2048.
I've tested it against the IBM DB2 & Postgres ODBC drivers and will continue to test and add support for all major databases. If you've got one you'd like to see let me know in the comments.
I've got plenty of improvements in the pipeline including:
-
DuckDB 0.7.0
It's not a dumb question at all. I'm pretty knowledgeable with DBs and still find it very difficult to understand how many of these front-end/pass-through engines work.
Checkout Postgres Foreign Data Wrappers. That might be the most well known approach for accessing one database through another. The Supabase team wrote an interesting piece about this recently.
https://supabase.com/blog/postgres-foreign-data-wrappers-rus...
You might also want to try out duckdb's approach to reading other DBs (or DB files). They talk about how they can "import" a sqlite DB in the above 0.7.0 announcement, but also have some other examples in their duckdblabs github project. Check out their "...-scanner" repos:
-
DuckDB – in-process SQL OLAP database management system
Doesn't postgres have a columnar option? If so, you could prob get better performance for your analytical interactions if you switched some tables to columnar.
Otherwise check out postgres scanner. https://github.com/duckdblabs/postgres_scanner
- DuckDB on YugabyteDB
-
Notes on the SQLite DuckDB Paper
DuckDB can actually read SQLite or Postgres directly! In the SQLite case, something like Litestream plus DuckDB could work really well!
Also, with Pyarrow's help, DuckDB can already do this with Delta tables!
-
Friendlier SQL with DuckDB
Interesting thought! I have not tried this yet so I only have a guess as an answer. Could you export the data as SQL statements and then run those statements on DuckDB? That may be easier to set up, but may take longer to run...
DuckDB also has the ability to read Postgres data directly, and there is a Postgres FDW that can read from DuckDB!
What are some alternatives?
dsq - Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.
odbc-scanner-duckdb-extension - A DuckDB extension to read data directly from databases supporting the ODBC interface
textql - Execute SQL against structured text like CSV or TSV
ClickBench - ClickBench: a Benchmark For Analytical Databases
roapi - Create full-fledged APIs for slowly moving datasets without writing a single line of code.
sqlite_scanner - DuckDB extension to read and write to SQLite databases
better-sqlite3 - The fastest and simplest library for SQLite3 in Node.js. [Moved to: https://github.com/WiseLibs/better-sqlite3]
budibase - Budibase is an open-source low code platform that helps you build internal tools in minutes 🚀
q - q - Run SQL directly on delimited files and multi-file sqlite databases
duckdb - DuckDB is an in-process SQL OLAP Database Management System
db-benchmark - reproducible benchmark of database-like ops