pg_ivm VS mssql-changefeed

FTA, because I don't like Jeopardy questions in headlines:

“Incremental View Maintenance (IVM) provides a method for keeping materialized views current by calculating and applying only the incremental changes, as opposed to the complete recomputation of contents performed by the REFRESH MATERIALIZED VIEW command.”

Article shows using the pg_ivm Postgres extension available here: https://github.com/sraoss/pg_ivm

There is an extension that provides some limited incremental MVIEW refresh: https://github.com/sraoss/pg_ivm

There’s awesome work being done on incremental view maintenance in postgres:

https://github.com/sraoss/pg_ivm

It makes sense to perform managed denormalization - use a materialized view or automatically refresh a table or foreign server (via FDW) using common triggers (like pg_ivm does). And it's fine to add a TTL to it and use as a read store... update on user login and make a partial index just for that. And that's how you could get CQRS...

We made mssql-changefeed for this purpose:

https://github.com/vippsas/mssql-changefeed

V1 requires to run a sweeper procedure in the background, but an upcoming v2 version does without the sweep loop. Unfortunately too fresh for readme to be updated, but relevant lines in tests to show it off:

https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/go/...

https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/go/...

https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/go/...

Library itself:

https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/mig...

Not OP, but there is an approach here of using a dedicated loop worker to assign post-commit ID sequence. I.e. using the outbox pattern once, simply to assign a post-commit ID.

https://github.com/vippsas/mssql-changefeed/blob/main/MOTIVA...

I wish DBs had this more built in, it seems a critical feature of a DB these days and the commit log already have very similar sequence numbers internally...

We do something similar, but in 2), instead of using the outbox pattern, we make use (in several different settings) of integers that are guaranteed to increment in commit order, then each consumer can track where their cursor is on the feed of changes. This requires some more coordination but it means that publishers of changes don't need one outbox per consumer or similar.

Then you can have "processes" that query for new data in an input table, and update aggregates/derived tables from that simply by "select * ... where ChangeSequenceNumber > @MaxSequenceNumberFromPreviousExecution"...

The idea here implemented for Microsoft SQL for the OLTP case:

https://github.com/vippsas/mssql-changefeed