RDS Database Migration Series - Facing The Giant: How we migrated 11 TB database

This page summarizes the projects mentioned and recommended in the original post on dev.to

Scout Monitoring - Performance metrics and, now, Logs Management Monitoring with Scout Monitoring
Get early access to Scout Monitoring's NEW Ruby logging feature [beta] by signing up now. Start for free and enable logs to get better insights into your Rails apps.
www.scoutapm.com
featured
InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
  • PaperTrail

    Track changes to your rails models

    The primary culprits were two tables (and their huge indexes) that contributed approximately 90% to the total size of the database. One of them was an audit trail (paper trail versions, to be exact), and the second one was more domain-specific for short-term rentals. It's a pre-computed cache of prices for properties depending on various conditions so that they don't need to be computed each time on the fly and can be easily distributed to other services.

  • Scout Monitoring

    Performance metrics and, now, Logs Management Monitoring with Scout Monitoring. Get early access to Scout Monitoring's NEW Ruby logging feature [beta] by signing up now. Start for free and enable logs to get better insights into your Rails apps.

    Scout Monitoring logo
  • ActiveRecord Import

    A library for bulk insertion of data into your database using ActiveRecord.

    At that point, we had to figure out some custom solution, as we didn't want to change the migration strategy from just the full load to the combination of full load and CDC. The good news was that we had already been using something that could be very useful in designing a custom solution - performing a bulk insert (using activerecord-import) of the archived records. It proved to be fast enough to restore a significant number of records. Also, nothing was preventing us from having a way higher parallelization degree than DMS. This could be our solution.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • historical data and "point in time" data modeling techniques, advice.

    1 project | /r/dataengineering | 28 Jun 2023
  • Best rails tools to automatically handle logging of things like all a user's actions, or changes to a record in a module - primarily for audit purposes.

    3 projects | /r/ruby | 9 Mar 2023
  • Best rails tools to automatically handle logging of things like all a user's actions, or changes to a record in a module - primarily for audit purposes.

    4 projects | /r/rails | 9 Mar 2023
  • Inventory/Sales Management module built on a Rails app - what would be the best way to "version" updates made against an SKU.

    1 project | /r/rails | 27 Dec 2022
  • is there a gem for tracking adhoc rails console changes

    2 projects | /r/rails | 15 Dec 2022

Did you konow that Ruby is
the 12th most popular programming language
based on number of metions?