pgsink VS debezium-examples

This is a really awesome project. I’ve used it on https://github.com/lawrencejones/pgsink to generate type safe bindings to the Postgres catalog tables, along with a few of the tables the project maintains itself.

pgx is really good: it's what I used to write logical decoders in https://github.com/lawrencejones/pgsink

It used to be that companies would write their own hacky scripts to perform this extraction - I've had terrible incidents caused by ETL database triggers in the past, and even built a few generic ETL tools myself.

Postgres change-capture device that supports high-throughput and low-latency capture to a variety of sinks (at first release, just Google BigQuery):

https://github.com/lawrencejones/pgsink

I know there's debezium and Netflix's dblog, but this project aims to be much simpler.

Forget about kafka and any other dependency: just point it at Postgres, and your data will be pushed into BigQuery. And for people with highly-performance-sensitive databases, the read workload has been designed with Postgres efficiency in mind.

I'm hoping pgsink could be a gateway drug to get small companies up and running with a data warehouse. If your datastore of choice is Postgres, it's a huge help to replicate everything into an analytics datastore. A similar tool has helped my company extract expensive work out of our primary database, which is super useful for scaling.

The project is 90% there, about 10hrs and some testing away from being useable. Once there, I'll be hitting up some start-up friends and seeing if they want to give it a whirl.

There are a lot of great example here -> https://github.com/debezium/debezium-examples

I'm thinking about open sourcing the framework that I wrote to implement a managed saga inspired to the framework package in Debezium saga order-service: https://github.com/debezium/debezium-examples/tree/main/saga

Check out their examples repo: https://github.com/debezium/debezium-examples/tree/main/debezium-server-postgres-pubsub

Hi, I am struggling with debezium and elastic search the problem seems to be on the elasticsearch side or docker. I am using this example and when I try to create an index with mappings the return is (52) Empty reply from server. Even when I run es container according to es documentation and make curl -X GET "localhost:9200/_cat/nodes?v=true&pretty" request server returns an empty reply.

We have some Kafka Streams related examples in the debezium-examples repo, e.g. for FK joins, aggregations, and using a state store. They show usage of Kafka Streams in the context of CDC, but the same patterns are more generically applicable of course.

Well, it works with Postgres, but apparently not CloudSQL for PostgreSQL.

The original code for the connector context can be found here: https://github.com/debezium/debezium-examples/tree/master/end-to-end-demo/debezium-jdbc