pgroll VS migra

My thoughts exactly. It's surprising that external online schema change tools for Postgres have only become a thing fairly recently! The only two I'm aware of are:

* pgroll: Written in Golang, first commits June 2023. https://github.com/xataio/pgroll

* pg-osc: Written in Ruby, first commits Dec 2021. https://github.com/shayonj/pg-osc

Meanwhile over in the MySQL and MariaDB ecosystem, external OSC tools have been around for quite some time, starting with oak-online-alter-table over 15 years ago. The most popular options today are pt-online-schema-change or gh-ost, but other battle-tested solutions include fb-osc, LHM, and the latest entry Spirit.

I thought I recognized xataio - they submitted pgroll a few months back https://news.ycombinator.com/item?id=37752366 (https://github.com/xataio/pgroll - Apache 2)

At the start of October we released pgroll, an open source tool for zero-downtime, reversible schema migrations for Postgres.

If you have any suggestions or questions, please open an issue in our GitHub repo, reach out to us on Discord or follow us on X / Twitter. We'd love to hear from you and keep you up to date with the latest progress on pgroll.

Any pgroll operations[0] that require a change to an existing column, such as adding a constraint, will create a new copy of the column and backfill it using 'up' SQL defined in the migration and apply the change to that new column.

There are no operations that will modify the data of an existing column in-place, as this would violate the invariant that the old schema must remain usable alongside the new one.

[0] - https://github.com/xataio/pgroll/tree/main/docs#operations-r...

This is a fantastic article! It shows that even simple migrations (like adding or removing a column) can be quite tricky to deploy in concert with the application deployement.

We (at Xata) have tried for a while to come up with a generic schema migration system for PostgreSQL that makes this easier. We ended up using views and temporary columns in such a way that we can provide both the "old" and the "new" schema simultaneously. Up/down triggers convert newly inserted data from old to new and the other way around. This also has the advantage the it can do rollbacks instantly by just dropping the "new" view.

We were just planning to announce this as an open source project this week, but actually it is already public, so if you are curious: https://github.com/xataio/pgroll

Forr postgres, how does the schema diffing aspect compare to migra?

https://github.com/djrobstep/migra

I'm asking because, although migra is excellent and there are multiple migrations tools based on it (at least https://github.com/bikeshedder/tusker and https://github.com/blainehansen/postgres_migrator), issues are piling up but development seem to be slowing down

We’ve extended the CLI migration feature and added Dashboard support. Database migrations give you a way to update your database using version-controlled SQL files. We’ve built a lot of tooling around our migrations, including reparation, migration cleanup using the squash command, and diffing (using migra) to generate a new migration or to detect schema drift.

I've had a lot of success with: https://github.com/djrobstep/migra

Is this similar to migra? There's a tool written in Rust that calls it, postgres_migrator (there's also tusker)

If you wish to auto-generate migrations, there are declarative schema change tools available for most relational databases. I'm the creator of Skeema [1] which provides them for MySQL, but there are options for other DBs too [2][3][4].

Prisma's migration system actually partially copied Skeema's design, while giving credit in a rather odd fashion which really rubbed me the wrong way: "The workflow of working with temporary databases and introspecting it to determine differences between schemas seems to be pretty common, this is for example what skeema does." [5]

While I doubt I was the first person to ever use that technique, I absolutely didn't copy it from anywhere, and it was never "pretty common". I'm not aware of any other older schema change systems that work this way.

[1] https://www.skeema.io

[2] https://github.com/djrobstep/migra

[3] https://github.com/k0kubun/sqldef

[4] https://david.rothlis.net/declarative-schema-migration-for-s...

[5] https://github.com/prisma/prisma-engines/blob/6be410e/migrat...

The best solution I've ever seen is this Rust library https://github.com/cornucopia-rs/cornucopia

You write plain SQL for you schema (just a schema.sql is enough) and plain SQL functions for your queries. Then it generates Rust types and Rust functions from from that. If you don't use Rust, maybe there's a library like that for your favorite language.

Optionally, pair it with https://github.com/bikeshedder/tusker or https://github.com/blainehansen/postgres_migrator (both are based off https://github.com/djrobstep/migra) to generate migrations by diffing your schema.sql files, and https://github.com/rust-db/refinery to perform those migrations.

Now, if you have simple crud needs, you should probably use https://postgrest.org/en/stable/ and not an ORM. There are packages like https://www.npmjs.com/package/@supabase/postgrest-js (for JS / typescript) and probably for other languages too.

If you insist on an ORM, the best of the bunch is prisma https://www.prisma.io/ - outside of the typescript/javascript ecosystem it has ports for some other languages (with varying degrees of completion), the one I know about is the Rust one https://prisma.brendonovich.dev/introduction

I really liked the idea proposed in https://github.com/djrobstep/migra but haven’t used it yet.

I'd love to be able to be able to declaratively make changes directly in the table create commands instead of manually creating new migration scripts every time. I've found migra (we use PostgreSQL) and it seems to be exactly what I'm looking for. I'm curious about other people's experience and why things like Migra are the norm.

Migra should do it https://databaseci.com/docs/migra