-
debezium
Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.
-
InfluxDB
InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.
-
> Logical replication is also great for replicating to other systems - for example Debezium [1] that writes all changes to a Kafka stream.
So I'm looking for a Rust library (or a C library) that receives a Postgres logical replication stream and is compatible with Postgres 15, does you know of something?
Debezium is unfortunately written in Java, and I think your Powersync is proprietary right? (I didn't find a Github link). I can find stuff like https://www.npmjs.com/package/pg-logical-replication but again, it's Javascript.
-
For C there should be good options.
For Rust it doesn't appear that well-supported.
A very simple approach is to poll for changes using `pg_logical_slot_get_changes()` - that should work with any driver. That's what I used for my initial experimentation, before switching over to the streaming replication protocol for better performance.
The streaming replication protocol is not that complicated, but currently you'll have to handle some of the low-level protocol yourself, or work with some very experimental implementations. There's a project to help get you started at [1], and some more discussion at [2].
For the logical decoder, wal2json is quite nice to experiment with, but I've found pgoutput is not that complicated and gives you something closer to the raw data.
[1]: https://github.com/seddonm1/logicaldecoding/
[2]: https://github.com/sfackler/rust-postgres/issues/116
-
For C there should be good options.
For Rust it doesn't appear that well-supported.
A very simple approach is to poll for changes using `pg_logical_slot_get_changes()` - that should work with any driver. That's what I used for my initial experimentation, before switching over to the streaming replication protocol for better performance.
The streaming replication protocol is not that complicated, but currently you'll have to handle some of the low-level protocol yourself, or work with some very experimental implementations. There's a project to help get you started at [1], and some more discussion at [2].
For the logical decoder, wal2json is quite nice to experiment with, but I've found pgoutput is not that complicated and gives you something closer to the raw data.
[1]: https://github.com/seddonm1/logicaldecoding/
[2]: https://github.com/sfackler/rust-postgres/issues/116
