pgroll VS omnigres

My thoughts exactly. It's surprising that external online schema change tools for Postgres have only become a thing fairly recently! The only two I'm aware of are:

* pgroll: Written in Golang, first commits June 2023. https://github.com/xataio/pgroll

* pg-osc: Written in Ruby, first commits Dec 2021. https://github.com/shayonj/pg-osc

Meanwhile over in the MySQL and MariaDB ecosystem, external OSC tools have been around for quite some time, starting with oak-online-alter-table over 15 years ago. The most popular options today are pt-online-schema-change or gh-ost, but other battle-tested solutions include fb-osc, LHM, and the latest entry Spirit.

I thought I recognized xataio - they submitted pgroll a few months back https://news.ycombinator.com/item?id=37752366 (https://github.com/xataio/pgroll - Apache 2)

At the start of October we released pgroll, an open source tool for zero-downtime, reversible schema migrations for Postgres.

If you have any suggestions or questions, please open an issue in our GitHub repo, reach out to us on Discord or follow us on X / Twitter. We'd love to hear from you and keep you up to date with the latest progress on pgroll.

Any pgroll operations[0] that require a change to an existing column, such as adding a constraint, will create a new copy of the column and backfill it using 'up' SQL defined in the migration and apply the change to that new column.

There are no operations that will modify the data of an existing column in-place, as this would violate the invariant that the old schema must remain usable alongside the new one.

[0] - https://github.com/xataio/pgroll/tree/main/docs#operations-r...

This is a fantastic article! It shows that even simple migrations (like adding or removing a column) can be quite tricky to deploy in concert with the application deployement.

We (at Xata) have tried for a while to come up with a generic schema migration system for PostgreSQL that makes this easier. We ended up using views and temporary columns in such a way that we can provide both the "old" and the "new" schema simultaneously. Up/down triggers convert newly inserted data from old to new and the other way around. This also has the advantage the it can do rollbacks instantly by just dropping the "new" view.

We were just planning to announce this as an open source project this week, but actually it is already public, so if you are curious: https://github.com/xataio/pgroll

I've been writing Postgres extensions for a few years now (mostly in C and Python), and I have to say that an important contribution from companies like Tembo and Omnigres is in the area of tooling and boilerplate. From workflows to Dockerfiles [0], that's important for the future of Postgres.

This has been discussed in the past [1], but the Postgres tooling ecosystem has been primarily C-Makefile—mailing list driven, and there used to be a lot of Makefile targets copy-pasting. Whenever major Postgres providers wanted to open source some of their extensions / sub-products. I still feel, however, that a lot of Postgres C-know-how is being slowly forgotten / lost, and I think it will be necessary again soon. Internal things as how Postgres handles varlena, StringInfo, JsonbValue, etc. The core abstractions that make Postgres work.

0: https://github.com/omnigres/omnigres/blob/master/Dockerfile

If you think that's cool, you might also want to check out Omnigres:

https://github.com/omnigres/omnigres

At Omnigres, our north star is to enable developers to laser-focus on business needs instead of fighting technological challenges.

We're fighting the complexity and inefficiencies of contemporary stacks by removing them instead of hiding them.

At the core, we are turning Postgres into an Application Runtime. Why? Because we believe that code and data are inseparable in pretty much all of the line-of-business application systems. Turns out, when done this way, applications work a lot faster, require a lot less maintenance and are simply easier to write.

Our foundation is open source and is available at https://github.com/omnigres/omnigres

We're backed by some great early-stage VCs and looking to onboard people who can move quickly, learn on the go and maintain the focus on the goals. Another way to look at it: we want to meet other pragmatic idealists.

Email: [email protected]

Check out Omnigres: https://github.com/omnigres/omnigres/wiki/Bounties

They just launched a bounty program recently.

Omnigres | Founding Engineer | SF Bay Area HQ | REMOTE

At Omnigres, our north star is to enable developers to laser-focus on business needs instead of fighting technological challenges.

We're fighting the complexity and inefficiencies of contemporary stacks by removing them instead of hiding them.

At the core, we are turning Postgres into an Application Runtime. Why? Because we believe that code and data are inseparable in pretty much all of the line-of-business application systems. Turns out, when done this way, applications work a lot faster, require a lot less maintenance and are simply easier to write.

Our foundation is open source and is available at https://github.com/omnigres/omnigres

We're backed by some great early-stage VCs and looking to onboard people who can move quickly, learn on the go and maintain the focus on the goals. Another way to look at it: we want to meet other pragmatic idealists.

You can apply here https://wellfound.com/jobs/2832133-founding-engineer

Couple of questions (with notes): Yes, I'd really like to write code next to the db with more suitable (for task) languages like Python. But at the moment SQL alone is supported..? Could one connect with Jupyter notebook somehow and have a REPL like experience with Omnigres instance?

Also, this [1] seems intriguing. How do containers connect to the db? What would the performance differences to the "internal" approach? Is this feature more like Lambda or for long running processes? Or something else? In any case very interesting.

Thank you!

[1] https://github.com/omnigres/omnigres/tree/master/extensions/...

Very much in the same vein: https://github.com/omnigres/omnigres