How is everyone ingesting backend relational data?

This page summarizes the projects mentioned and recommended in the original post on /r/dataengineering
MySQL Python mysql-replication-protocol mysql-binlog

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

  • python-mysql-replication

    Pure Python Implementation of MySQL replication protocol build on top of PyMYSQL

    • From backend relational tables to data warehouses my team has mostly relied on change data capture replication. We use MySQL upstream, and historically used AWS DMS or Attunity Replicate to replicate directly to SQL server. Recently we made the switch to Snowflake, and used mostly AWS DMS to replicate CDC data to S3 (lists individual inserts, updates, deletes), and then from there use snowpipes to copy to snowflake and then a job to merge that data into the target table to get the latest state. In addition we've used this library in production https://github.com/noplay/python-mysql-replication, and still use it today for one high volume, critical data source. Generally we see data go end to end in a matter of minutes, but occasionally there are spikes in latency.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Why Binlog size grows drastically when isolation level set to "Repeatable Read" & When isolation level set to "Read Committed" the size of Binlog file reduces ?

    1 project | /r/mysql | 21 Apr 2023

  • How to Use BinLogs to Make an Aurora MySQL Event Stream

    3 projects | dev.to | 3 Oct 2022

  • Robust data transfer mechanism?

    1 project | /r/learnpython | 24 Apr 2021

  • Merging data from multiple Excel files

    1 project | dev.to | 28 Mar 2024

  • Echolocate your MySQL health with real-time monitoring in the terminal

    1 project | news.ycombinator.com | 10 Mar 2024