Querying Postgres Tables Directly from DuckDB

• duckdb_fdw

  4 235 7.3 PLpgSQL

  DuckDB Foreign Data Wrapper for PostgreSQL

• duckdb

  52 16,749 10.0 C++

  DuckDB is an in-process SQL OLAP Database Management System

  

https://github.com/duckdb/duckdb/pull/2832

Does that help or do you have any other questions?

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• ClickHouse

  208 34,153 10.0 C++

  ClickHouse® is a free analytics DBMS for big data

  

It is very easy with ClickHouse. All you need is:

SELECT ... FROM postgresql(...) FORMAT Parquet

And you can run this query without installing ClickHouse, using the clickhouse-local command-line tool.

It can be downloaded simply as:

curl https://clickhouse.com/ | sh

• pqrs

  4 249 4.9 Rust

  Command line tool for inspecting Parquet files

This won't let you ETL from PG to parquet, but I used this in anger the other day https://github.com/manojkarthick/pqrs Worked quite well for my purposes!

• connector-x

  11 1,769 7.9 Rust

  Fastest library to load data from DB to DataFrames in Rust and Python

  

I was trying https://github.com/sfu-db/connector-x and hacking around with this https://github.com/spitz-dan-l/postgres-binary-parser but it turned out that a COPY to csv using asyncpg and then converting to parquet was the fastest.

• postgres-binary-parser

  2 6 0.0 Python

  Cython implementation of a parser for PostgreSQL's COPY WITH BINARY format

I was trying https://github.com/sfu-db/connector-x and hacking around with this https://github.com/spitz-dan-l/postgres-binary-parser but it turned out that a COPY to csv using asyncpg and then converting to parquet was the fastest.

Suggest a related project