rust-postgres VS mssql-changefeed

For C there should be good options.

For Rust it doesn't appear that well-supported.

A very simple approach is to poll for changes using `pg_logical_slot_get_changes()` - that should work with any driver. That's what I used for my initial experimentation, before switching over to the streaming replication protocol for better performance.

The streaming replication protocol is not that complicated, but currently you'll have to handle some of the low-level protocol yourself, or work with some very experimental implementations. There's a project to help get you started at [1], and some more discussion at [2].

For the logical decoder, wal2json is quite nice to experiment with, but I've found pgoutput is not that complicated and gives you something closer to the raw data.

[1]: https://github.com/seddonm1/logicaldecoding/

[2]: https://github.com/sfackler/rust-postgres/issues/116

This is a bit of a rant so please bear with me. I wrote a small utility program a long time ago that used this version of the postgres crate

En este post aprendermos a usar Rust, Tonic y la crate gRPC, y implementaremos un CRUD con Postgresql database.

I can understand that this can be frustrating and I know that the situation there is not ideal for diesel. There are certainly things to improve there by either providing a bundling support which builds the native library as part of the normal build process or by implementing a pure rust connection implementation. Both is possible with diesel, but requires some work. At least the pure rust connection implementation is something that can be provided by a third party crate now with upcoming diesel 2.0 release. If you are interested in that checkout this and this issue. As for the bundling support: This requires changes in the mysqlclient-sys and pq-sys crates. Again help there is welcome. In the end it makes me sad that some people have repeating decided that a solution to this problem is to write just another crate instead of helping to fix these issues. This just results in everyone have more work to do, as there are now two non-perfect solutions instead of having one slightly improved solution.

We created this little library to have a fully native type for Postgres Numerics with the rust-postgres (https://github.com/sfackler/rust-postgres) library.

The problem with a crate like postgres is that you have to define the types of the query at compile-time. And if you use the simple query protocol in postgres, you just get a bunch of strings, i.e. no proper typing at all. However, for maximal flexibility arbitrary queries should work in pigeon, without knowing the database schema.

Some nifty things about usual: - It's a generic wrapper over any SQL "row" object. The first implementation that's provided is for tokio-postgres, but traits are available to implement over whatever you'd like. - It provides static typing for partial queries. That is, it supports fetching a subset of fields from a row and makes a unique type for the return value. This gives you some neat-o type safety like this:

In this post, we will learn how to use Rust Tonic gRPC crate. We will learn how to implement CRUD with Postgresql database.

(see: https://github.com/sfackler/rust-postgres/blob/e15c9b1415f69821799f1370246581c1600a6196/postgres-protocol/src/types/mod.rs#L137)

We made mssql-changefeed for this purpose:

https://github.com/vippsas/mssql-changefeed

V1 requires to run a sweeper procedure in the background, but an upcoming v2 version does without the sweep loop. Unfortunately too fresh for readme to be updated, but relevant lines in tests to show it off:

https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/go/...

https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/go/...

https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/go/...

Library itself:

https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/mig...

Not OP, but there is an approach here of using a dedicated loop worker to assign post-commit ID sequence. I.e. using the outbox pattern once, simply to assign a post-commit ID.

https://github.com/vippsas/mssql-changefeed/blob/main/MOTIVA...

I wish DBs had this more built in, it seems a critical feature of a DB these days and the commit log already have very similar sequence numbers internally...

We do something similar, but in 2), instead of using the outbox pattern, we make use (in several different settings) of integers that are guaranteed to increment in commit order, then each consumer can track where their cursor is on the feed of changes. This requires some more coordination but it means that publishers of changes don't need one outbox per consumer or similar.

Then you can have "processes" that query for new data in an input table, and update aggregates/derived tables from that simply by "select * ... where ChangeSequenceNumber > @MaxSequenceNumberFromPreviousExecution"...

The idea here implemented for Microsoft SQL for the OLTP case:

https://github.com/vippsas/mssql-changefeed