Pulling CDC data from Postgres

• conduit

  7 342 9.4 Go

  Conduit streams data between data stores. Kafka Connect replacement. No JVM required. (by ConduitIO)

  

I'd like to mention Conduit + its Postgres connector. The Pg connector comes built-in, so all that is needed is a single Conduit binary to get started. It relies on WAL, but the connector creates the replication slot itself (if needed).

• conduit-connector-postgres

  1 9 6.9 Go

  Conduit connector for PostgreSQL

  

I'd like to mention Conduit + its Postgres connector. The Pg connector comes built-in, so all that is needed is a single Conduit binary to get started. It relies on WAL, but the connector creates the replication slot itself (if needed).

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• dozer

  20 1,446 9.7 Rust

  Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks. (by getdozer)

  

You can consider using Dozer (https://github.com/getdozer/dozer), an open-source data API backend. Dozer has a PostgreSQL connector that you can use to monitor any changes in your database schema and data. Although Dozer is not specifically designed for ETL tasks, you can still use it for this purpose by using Lambda functions. Dozer provides a Lambda runtime to execute user-defined functions. So you can use PostgreSQL connector ( to capture CDC) with Lambda runtime to load data to snowflake.

• plumber

  19 2,041 7.7 Go

  A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.

  

I recommend Streamdal. The connecting agent is open source and distributed by default, so it will scale horizontally WAY better than Debezium. All data ingested is indexed into parquet as well, and you can do serverless functions/transforms on the platform to reduce Snowflake compute costs.

• conduit-connector-oracle

  1 0 7.8 Go

  Conduit connector for Oracle

  

The Oracle connector is actually done, and there's both, a source and a destination connector for it: https://github.com/conduitio-labs/conduit-connector-oracle

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project