DuckDB: Querying JSON files as if they were tables

• dsq

  20 3,603 4.3 Go

  Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.

  

Welcome to the gang! :)

https://github.com/multiprocessio/dsq#comparisons

• octosql

  34 4,689 4.3 Go

  OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.

This is really cool!

With their Postgres scanner[0] you can now easily query multiple datasources using SQL and join between them (i.e. Postgres table with JSON file). Something I strived to build with OctoSQL[1] before.

It's amazing to see how quickly DuckDB is adding new features.

Not a huge fan of C++, which is right now used for authoring extensions, it'd be really cool if somebody implemented a Rust extension SDK, or even something like Steampipe[2] does for Postgres FDWs which would provide a shim for quickly implementing non-performance-sensitive extensions for various things.

Godspeed!

[0]: https://duckdb.org/2022/09/30/postgres-scanner.html

[1]: https://github.com/cube2222/octosql

[2]: https://steampipe.io

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• steampipe

  146 6,366 9.7 Go

  Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

  

This is really cool!

With their Postgres scanner[0] you can now easily query multiple datasources using SQL and join between them (i.e. Postgres table with JSON file). Something I strived to build with OctoSQL[1] before.

It's amazing to see how quickly DuckDB is adding new features.

Not a huge fan of C++, which is right now used for authoring extensions, it'd be really cool if somebody implemented a Rust extension SDK, or even something like Steampipe[2] does for Postgres FDWs which would provide a shim for quickly implementing non-performance-sensitive extensions for various things.

Godspeed!

[0]: https://duckdb.org/2022/09/30/postgres-scanner.html

[1]: https://github.com/cube2222/octosql

[2]: https://steampipe.io

• duckdb-extension-framework

  1 54 1.3 Rust

To answer myself, I've found a project which enables extension development for DuckDB using Rust[0].

[0]: https://github.com/Mause/duckdb-extension-framework

• trustfall

  24 2,263 9.3 Rust

  A query engine for any combination of data sources. Query your files and APIs as if they were databases!

• ibis

  22 4,041 10.0 Python

  the portable Python dataframe library

  

• duckdb

  52 16,356 10.0 C++

  DuckDB is an in-process SQL OLAP Database Management System

  

Thanks for trying it out!

Could you perhaps open an issue [1] or share the file with us so we could investigate the problem?

[1] https://github.com/duckdb/duckdb/issues

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• nba-monte-carlo

  3 335 9.4 Python

  Monte Carlo simulation of the NBA season, leveraging dbt, duckdb and evidence.dev

• ducker

  1 13 3.6 Dockerfile
  

I made a docker image with a number of extensions already installed and enabled so you can start using DuckDB with the lowest friction.

`alias dckr='docker run --rm -it -v $(pwd):/data -w /data duckerlabs/ducker'`

then `dckr` gives you a DuckDB shell with PRQL, httpfs, json, parquet, postgres, sqlite, and substrait enabled.

For example, to get the first 5 lines of a csv file named "albums.csv", you could run it with PRQL

```dckr -c 'from `albums.csv` | take 5;'```

https://github.com/duckerlabs/ducker

Suggest a related project