RDS Database Migration Series - Facing The Giant: How we migrated 11 TB database

This page summarizes the projects mentioned and recommended in the original post on dev.to

Scout Monitoring - Rennaisance engineers rejoice! 1 gem 5 min to app monitoring
5-minute onboarding. No sales team. Devs in the support channels. No DevOps team required. Get the free app insights every engineer deserves with Scout Monitoring.
www.scoutapm.com
featured
InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
  • PaperTrail

    Track changes to your rails models

  • The primary culprits were two tables (and their huge indexes) that contributed approximately 90% to the total size of the database. One of them was an audit trail (paper trail versions, to be exact), and the second one was more domain-specific for short-term rentals. It's a pre-computed cache of prices for properties depending on various conditions so that they don't need to be computed each time on the fly and can be easily distributed to other services.

  • ActiveRecord Import

    A library for bulk insertion of data into your database using ActiveRecord.

  • At that point, we had to figure out some custom solution, as we didn't want to change the migration strategy from just the full load to the combination of full load and CDC. The good news was that we had already been using something that could be very useful in designing a custom solution - performing a bulk insert (using activerecord-import) of the archived records. It proved to be fast enough to restore a significant number of records. Also, nothing was preventing us from having a way higher parallelization degree than DMS. This could be our solution.

  • Scout Monitoring

    Rennaisance engineers rejoice! 1 gem 5 min to app monitoring. 5-minute onboarding. No sales team. Devs in the support channels. No DevOps team required. Get the free app insights every engineer deserves with Scout Monitoring.

    Scout Monitoring logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • historical data and "point in time" data modeling techniques, advice.

    1 project | /r/dataengineering | 28 Jun 2023
  • Best rails tools to automatically handle logging of things like all a user's actions, or changes to a record in a module - primarily for audit purposes.

    3 projects | /r/ruby | 9 Mar 2023
  • Best rails tools to automatically handle logging of things like all a user's actions, or changes to a record in a module - primarily for audit purposes.

    4 projects | /r/rails | 9 Mar 2023
  • Inventory/Sales Management module built on a Rails app - what would be the best way to "version" updates made against an SKU.

    1 project | /r/rails | 27 Dec 2022
  • is there a gem for tracking adhoc rails console changes

    2 projects | /r/rails | 15 Dec 2022