Debezium-design-documents Alternatives

Similar projects and alternatives to debezium-design-documents

transfer

7 530 9.6 Go debezium-design-documents VS transfer

Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.
universal-data

1 8 1.7 Python debezium-design-documents VS universal-data

Lightweight ELT/ETL that's perfect for api data extraction and transformation.
InfluxDB

www.influxdata.com featured

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better debezium-design-documents alternative or higher similarity.

Suggest an alternative to debezium-design-documents

debezium-design-documents reviews and mentions

Posts with mentions or reviews of debezium-design-documents. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-07-24.

Launch HN: Artie (YC S23) – Real time data replication to data warehouses
4 projects | news.ycombinator.com | 24 Jul 2023

Thanks for the comment!
Your comment regarding DDL is interesting.
Today, this is what happens:
1/ Column doesn't exist in the destination, we'll create it based on our typing inference from the data type (important: not the data value).
2/ Certain tools will handle automatic column data type conversion if a change like this was detected at the source. We do not do this. We will simply hard fail and cause head-of-line blocking reasons being: this is anti-pattern and should be rare, in which case - it's okay to cause an err and require manual intervention for this breaking change.
3/ If the column has been dropped from the source, you as the end user can decide whether this column should be also dropped in the destination, or not. The default is not to drop it.
^ We hear more customers explicitly don't want columns to be dropped because it could cause downstream errors, such as other views / tables not compiling due to referencing a non-existent column.
We haven't heard much from folks that don't even want columns to be added. If there is a need, we can definitely add that as a config option to provide maximum configurability.
> Finally, the biggest issue with CDC always ends up being the seed loads, recoveries and the incremental snapshot strategies.
Yep totally. On the recovery bit, this is exactly why we are leveraging Kafka. If there are any particular issues, we simply don't commit the offset and cause head-of-line blocking.
On the incremental snapshot and recoveries bit, we primarily leverage Debezium's DDD-3 high watermark strategy [1] for MySQL and MongoDB. Postgres has a different issue in that replication slots can grow really fast, esp on AWS! [2]. We ended up writing our own custom snapshotter for Postgres that is Debezium compatible to onboard customers that have a massive dataset and cannot afford to have a read lock on their WAL.
[1] https://github.com/debezium/debezium-design-documents/blob/m...