Our great sponsors
-
prql
PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
SQL the language has many drawbacks.
My weapon of choice when using Python is SQLAlchemy; it gives you composability.
Also, there's https://github.com/PRQL/prql that compiles to SQL.
> I asked was about querying data without ever using a SQL language, like tapping directly into the data.
I agree (making https://tablam.org to try a fix & working on https://github.com/clockworklabs/SpacetimeDB in the SQL conformance).
Before I think SQL was bad. *Now I'm certain*. SQL is absurdly massive for things that could have collapse all the features 10x or more.
However, working in an RDBM now I also understand why is not desirable to make "raw" calls to the DB: The engine MUST mediate all the calls to make things works (from query optimization, execution, iteration, lock management, transaction management, etc).
Is incredible how much sophistication is in a simple `SELECT * FROM table`.
What I wish is to build a `Wasm-like` IR so that is what anybody target, and `SQL` is not the mediator.
> I asked was about querying data without ever using a SQL language, like tapping directly into the data.
I agree (making https://tablam.org to try a fix & working on https://github.com/clockworklabs/SpacetimeDB in the SQL conformance).
Before I think SQL was bad. *Now I'm certain*. SQL is absurdly massive for things that could have collapse all the features 10x or more.
However, working in an RDBM now I also understand why is not desirable to make "raw" calls to the DB: The engine MUST mediate all the calls to make things works (from query optimization, execution, iteration, lock management, transaction management, etc).
Is incredible how much sophistication is in a simple `SELECT * FROM table`.
What I wish is to build a `Wasm-like` IR so that is what anybody target, and `SQL` is not the mediator.
Started going really deep into SQLite.
Couple hiccups like making sure when creating tables to use strict mode and enabling foreign keys that every one needs to read about.
I exported about 8million rows from my Apple health records (about eight years for something’s like steps from before watch) and into a SQLite database from the xml dump using python (twice as fast a rust for some reason) in about 110 seconds. Dates are stored as Unix time stamps. Had ChatGPT generate some queries that showed my percent change in things like resting heart rate using LAG and AVG return in less than a second.
My job at work https://unsupervised.com/ does a lot of KPI work so I’ve been interesting in learning how to to do see how metrics correlate.
Nextjs server components and https://github.com/kysely-org/kysely also make working with SQLite a breeze.