temporal_tables VS connectors

It was reimplemented in pure SQL here https://github.com/nearform/temporal_tables for this purpose

I enjoyed this blog. I think it provides a great succinct overview of various approaches native to Postgres.

For the "capture changes in an audit table" section, I've had good experiences at a previous company with the Temporal Tables pattern. Unlike other major RDBMS vendors, it's not built into Postgres itself, but there's a simple pattern [1] you can leverage with a SQL function.

This allows you to see a table's state as of a specific point in time. Some sample use cases:

- "What was this user's configuration on Aug 12?"

- "How many records were unprocessed at 11:55pm last night?"

- "Show me the diff on feature flags between now and a week ago"

[1]: https://github.com/nearform/temporal_tables

One of these Postgres-based implementations of SQL:2011's temporal versioning features might get you close enough:

- https://github.com/nearform/temporal_tables

You don't really need to install an extension to use temporal tables, there is an alternative (https://github.com/nearform/temporal_tables) implemented purely as a plpgsql trigger so that it works everywhere.

I was part of a team at NearForm using this for a project on an EC2 instance. In order to move to AWS RDS we had to recreate the functionality of temporal_tables as a PostgreSQL function, rather than extension.

When we switched, we found that although there were minor bugs, we didn't have any noticeable loss of performance and we have used it ever since for many projects.

https://github.com/nearform/temporal_tables

If you're also limited by cloud services and the extensions limitations, this is a great solution.

No. We implemented our own [1] for a few reasons:

* Scaling well to multi-TB DBs without pinning the write-ahead log (potentially filling your DB's disk) while the backfill is happening. Instead, our connector constantly reads the WAL and works well in setups like Supabase that have very restrictive WAL sizes (1GB iirc).

* Incremental fault-tolerant backfills that can be stopped and resumed at will.

* Being able to offer "precise" captures which are logically consistent in terms of the sequence of create/update/delete events.

The last one becomes really interesting when paired with REPLICA IDENTITY FULL, because you can feed it into an incremental computation (perhaps differential dataflow) for streaming updates of a continuous computation.

Our work is based off of the Netflix DBLog paper, which we took and ran with.

[1] https://github.com/estuary/connectors/tree/main/source-postg...

Our connectors themselves are fully OSS (for example, here's PostgreSQL)

We've used https://github.com/estuary/connectors/pkgs/container/source-... to load data sets in the many terabytes. Caveat that, while it's implemented to Airbyte's spec, we've only used it with Flow.