How Change Data Capture (CDC) Works with Streaming Database

• risingwave

  27 6,309 10.0 Rust

  Cloud-native SQL stream processing, analytics, and management. KsqlDB and Apache Flink alternative. 🚀 10x more productive. 🚀 10x more cost-efficient.

  

This demo leverages the following GitHub repository where we assume that all necessary things are set up using Docker compose. It will start multiple containers including MySQL with pre-populated inputs. You can see here what are the additional components in this Docker configuration.

• PostgreSQL

  408 14,734 10.0 C

  Mirror of the official PostgreSQL GIT repository. Note that this is just a *mirror* - we don't work with pull requests on github. To contribute, please see https://wiki.postgresql.org/wiki/Submitting_a_Patch

  

Change Data Capture is a technique used to capture and propagate changes made to a database such as MySQL, Microsoft SQL, Oracle, PostgreSQL, MongoDB, or Cassandra. CDC works by continuously monitoring the database for any changes made to the data. Multiple types of change data capture patterns can be used for data processing from a database. These include log-based CDC, trigger-based CDC, CDC based on timestamps, and difference-based CDC.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• MySQL

  146 10,270 9.8 C++

  MySQL Server, the world's most popular open source database, and MySQL Cluster, a real-time, open source transactional database.

  

Change Data Capture is a technique used to capture and propagate changes made to a database such as MySQL, Microsoft SQL, Oracle, PostgreSQL, MongoDB, or Cassandra. CDC works by continuously monitoring the database for any changes made to the data. Multiple types of change data capture patterns can be used for data processing from a database. These include log-based CDC, trigger-based CDC, CDC based on timestamps, and difference-based CDC.

• ApacheKafka

  104 28 0.0

  A curated re-sources list for awesome Apache Kafka

A streaming database is a type of database that is designed to handle continuous data streams in real-time and makes it possible to query this data. You can read more about how a Streaming database differs from a Traditional database and how to choose the right streaming database in my other blog posts. CDC is particularly useful when working with streaming databases, you can ingest CDC data from directly databases (See an example in the next section) without setting up additional services like Kafka.

• debezium

  80 9,884 9.9 Java

  Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.

  

If you’re already using Debezium to extract CDC logs into Kafka, you can just set up RisingWave to consume changes from that Kafka topic. In this case, Kafka acts like a hub of CDC data, and beside RisingWave, other downstream systems like search index or data warehouses can consume changes as well.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project