Using SQL inside Python pipelines with Duckdb, Glaredb (and others?)

• duckdb

  52 16,576 10.0 C++

  DuckDB is an in-process SQL OLAP Database Management System

  

Duckdb: https://github.com/duckdb/duckdb - seems pretty popular, been keeping an eye on this for close to a year now.

• glaredb

  6 519 9.8 Rust

  GlareDB: An analytics DBMS for distributed data

  

Glaredb: https://github.com/GlareDB/glaredb - just heard about this last week. We played around with hooking directly into snowflake, so that was cool, but I haven't heard of anyone else using it.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• techslamneggs

  1 0 3.7 Python

  The code for my May 3, 2023 workshop at Greenville's Tech Slam 'N Eggs!

Let me know if you have any questions about how we implement our ingestions, kind of like this demo project. You may also find this video useful.

• Meerschaum

  17 121 7.8 Python

  Create and manage data pipes with Meerschaum.

This sounds like a great use case for Meerschaum. You can organize your scripts into plugins and build out incremental transformations in SQL. We use Meerschaum Compose for client integrations and ETL in a similar workflow to yours.

• risingwave

  27 6,283 10.0 Rust

  Cloud-native SQL stream processing, analytics, and management. KsqlDB and Apache Flink alternative. 🚀 10x more productive. 🚀 10x more cost-efficient.

  

How real-timey are you projects. If more streams, check out rising wave too.

• chdb

  18 1,691 9.6 C++

  chDB is an embedded OLAP SQL Engine 🚀 powered by ClickHouse

  

New kid on the block that I prefer over DuckDB is CHDB (https://github.com/chdb-io/chdb). Embedded ClickHouse, so once you out grow your laptop you can simply move to an actual OLAP that's Open-source.

Suggest a related project