vdsql
quack-reduce
vdsql | quack-reduce | |
---|---|---|
1 | 2 | |
64 | 122 | |
- | 9.0% | |
2.9 | 4.8 | |
10 months ago | 4 months ago | |
Python | Python | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
vdsql
-
Is there a CLI interface to browse SQL databases?
I would especially keep an eye on the `vdsql` project, a VisiData plugin that uses `Ibis` to harness the power of SQL: https://github.com/visidata/vdsql =) It just had its first release.
quack-reduce
- quack-reduce: duckdb as a stateless query engine over a data lake
-
Summing columns in remote Parquet files using DuckDB
We can run a DuckDb instance (EC2/S3) closer to the data so that sorta helps too.
What I'm really excited about using DuckDB in a similar way to map-reduce. What if there was a way to take some SQL's logical plan and turn it into a physical plan that uses compute resources from a pool serverless DuckDB instances. Starting at the leafs of the graph (physical plan) pulling data from the source (parquet), and returning their completed work up the branches, until it is completed and ready to be used as the results.
I've seen a few examples of this already, but nothing that I would consider production ready. I have a hunch that someone is going to drop such a project on us shortly, and it's going to change a lot of things we have become use to in the data world.
https://github.com/BauplanLabs/quack-reduce
What are some alternatives?
inline-sql - 🪄 Inline SQL in any Python program
parquet-format - Apache Parquet
gobang - A cross-platform TUI database management tool written in Rust
sqlglot - Python SQL Parser and Transpiler
sql_to_ibis - A Python package that parses sql and converts it to ibis expressions
ibis - the portable Python dataframe library
influxdb3-python - Python module that provides a simple and convenient way to interact with InfluxDB 3.0.
icedb - An in-process Parquet merge engine for better data warehousing in S3
airflow-elt-blueprint - A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.