safeql VS postgres_migrator

I want sum types.

I want a statically-typed way of constructing composable queries that follow SQL rather than reinvent a different thing. It doesn't have to be the same syntax but it has to be the same structuring.

I started writing one[0] and stopped before doing all the boilerplate code generation, having moved on from the JVM ecosystem for the time being. One thing it does is treat most things like sets so we don't end up with N+1 queries. Another trick it uses is collapsing constant expressions via an expression evaluation library[1].

[0] https://github.com/karmakaze/safeql

[1] https://github.com/karmakaze/moja

Over the years, I've written many apps and utilities for myself or others (that didn't end up get used). These are the interesting ones I remember. Many not quite complete/usable. Other than hackerer.news none of them are 'up' and running. Some have and others haven't been published as opensource.

- https://hackerer.news HN viewer (source[0]): I use daily so I can see today's top stories in reverse chronological order with mainstream topics sorted to the bottom.

- qwickly[1] keyboard layout: I use all the time as an easier to learn and more comfortable to type than Colemak/Tarmak

- safeql[2]: Java type-safe SQL expression composer that reduces constant expressions and eliminates N+1 queries loading associations by always operating on set relation or array of models.

- moja[3]: Composable computation pipelines for Java: Async, Lazy, Option, Try, Result, Multi (List), Stated, Reader, Logger, Writer.

- gitgrep.com[4] Opensource SaaS version of etsy/houndd (now called hound-search).

- statuspages.me: Status page aggregator with dynamic javascript for scraping each source using selector expressions.

- movies to watch aggregator: with links to sources to watch. It was hard then to get 3rd party deep links into streaming sites so included some torrent links. Got a DMCA phone call, so took it down. Combined thumbnails, summaries, actors(?), imdb ratings, links.

- java2cpp: Translate a moderately sized java app with test suite to c++, not 100% required final manual fixups.

- swift2java (or maybe it was java2swift, it's fuzzy now): translate Swift to Java obviously, using ANTLR4. Not 100% required final manual fixups.

- gui2log: to make an ASCII rendition of on-screen GUI widgets into an application log file when form submitted, so users couldn't complain that they saw X, but got Y.

- some basic stats/ML algorithms: k-nearest neighbour, RNN back-propagation, etc?

- Java in-memory DB: Small SQL-like memory tables with indexing/searching.

- wwwsqldesigner: This exists as opensource and I extended it to infer foreign key relationships based on naming conventions used in a MySQL schema. It was great for zooming around a large ERD.

- tracelog: combination of microservices parent/child span logging and generated high level events shown as a sequence diagram. Integrated with Loggly for full/verbose logs of selected high-level events.

- pcl2bmp downscaler: Reduce high resolution HP LaserJet (PCL5) printed to file to lower resolution bitmap pages for screen display (before retina DPI was common). It aimed to shrink same-color areas and preserve black/white transitions while reducing.

[0] https://gitlab.com/karmakaze/hackerer-news

[1] https://github.com/qwickly-org/Qwickly

[2] https://github.com/karmakaze/safeql

[3] https://github.com/karmakaze/moja

[4] https://github.com/gitgrep-com/gitgrep

I completely agree. I pretty much stopped using Spring/Boot because of it, even though it could be used without Hibernate/JPA.

I tried sql2o and later switched to jdbi and Javalin for a lightweight framework. I started making a typesafe library[0] that maps bottom-up like SQL expressions but development as stalled as I haven't been doing much side-project work to use it.

[0] https://github.com/karmakaze/safeql

Interesting choice of JDBI. I was working on an SQL-friendly ORM[0] also due to distaste with Hibernate/JPQL and chose JDBI, not because it was great in any way but it did what I needed and not much else. What influenced your choice and were there any close runner-ups?

[0] https://github.com/karmakaze/safeql

Thanks! Yeah definitely agree that building out declarative table management for Postgres would be a major effort. A few open source projects I've seen in that area include:

https://github.com/sqldef/sqldef (Go)

https://github.com/bikeshedder/tusker (Python but being ported to Rust)

https://github.com/tyrchen/renovate (Rust)

https://github.com/blainehansen/postgres_migrator (Rust)

Some of these are based on parsing SQL, and others are based on running the CREATEs in a temporary location and introspecting the result.

The schema export side can be especially tricky for Postgres, since it lacks a built-in equivalent to MySQL's SHOW CREATE TABLE. So most of these declarative pg tools shell out to pg_dump, or require the user to do so. But sqldef actually implements CREATE TABLE dumping in pure Golang if I recall correctly, which is pretty cool.

There's also the question of implementing the table diff logic from scratch, vs shelling out to another tool or using a library. For the latter path, there's a nice blog post from Supabase about how they evaluated the various options: https://supabase.com/blog/supabase-cli#choosing-the-best-dif...

Forr postgres, how does the schema diffing aspect compare to migra?

https://github.com/djrobstep/migra

I'm asking because, although migra is excellent and there are multiple migrations tools based on it (at least https://github.com/bikeshedder/tusker and https://github.com/blainehansen/postgres_migrator), issues are piling up but development seem to be slowing down

Is this similar to migra? There's a tool written in Rust that calls it, postgres_migrator (there's also tusker)

The best solution I've ever seen is this Rust library https://github.com/cornucopia-rs/cornucopia

You write plain SQL for you schema (just a schema.sql is enough) and plain SQL functions for your queries. Then it generates Rust types and Rust functions from from that. If you don't use Rust, maybe there's a library like that for your favorite language.

Optionally, pair it with https://github.com/bikeshedder/tusker or https://github.com/blainehansen/postgres_migrator (both are based off https://github.com/djrobstep/migra) to generate migrations by diffing your schema.sql files, and https://github.com/rust-db/refinery to perform those migrations.

Now, if you have simple crud needs, you should probably use https://postgrest.org/en/stable/ and not an ORM. There are packages like https://www.npmjs.com/package/@supabase/postgrest-js (for JS / typescript) and probably for other languages too.

If you insist on an ORM, the best of the bunch is prisma https://www.prisma.io/ - outside of the typescript/javascript ecosystem it has ports for some other languages (with varying degrees of completion), the one I know about is the Rust one https://prisma.brendonovich.dev/introduction