Our great sponsors
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
python-mysql-replication
Pure Python Implementation of MySQL replication protocol build on top of PyMYSQL
ℹ️ I tried many different ways of getting a reliable binlog stream using typescript libraries and failed. I tried https://github.com/nevill/zongji, mysql2 (which didn't actually use the ROW format) and hacking on the normal mysql npm library myself. This python module which the AWS Blog: Streaming Changes in a Database with Amazon Kinesis uses was much faster to get going.
The BinLogStreamReader has several inputs that we need to retrieve. First we'll retrieve the cluster's secret with the database host/username/password and then we'll fetch the serverId we stored in S3.
Related posts
- Why Binlog size grows drastically when isolation level set to "Repeatable Read" & When isolation level set to "Read Committed" the size of Binlog file reduces ?
- How is everyone ingesting backend relational data?
- Robust data transfer mechanism?
- Merging data from multiple Excel files
- Echolocate your MySQL health with real-time monitoring in the terminal