ActiveRecord Import VS PaperTrail

At that point, we had to figure out some custom solution, as we didn't want to change the migration strategy from just the full load to the combination of full load and CDC. The good news was that we had already been using something that could be very useful in designing a custom solution - performing a bulk insert (using activerecord-import) of the archived records. It proved to be fast enough to restore a significant number of records. Also, nothing was preventing us from having a way higher parallelization degree than DMS. This could be our solution.

I've used the activerecordimport gem with great success

If you're running on Rails 5 and older versions, you might face the situtation where writing raw customized SQL or using gems like activerecord-import is the final option you left with.

In the first step, we only validate if the information is all there and correctly formatted. For example, if a location name is missing, we highlight that error in the spreadsheet. If everything looks good, we save the locations to the database using the awesome Activerecord-Import library, but skip calling the geocoding API. Rather, we mark these locations as "to be geocoded".

Have you tried Rails Import: https://github.com/zdennis/activerecord-import

1. ACTIVERECORD IMPORT

The primary culprits were two tables (and their huge indexes) that contributed approximately 90% to the total size of the database. One of them was an audit trail (paper trail versions, to be exact), and the second one was more domain-specific for short-term rentals. It's a pre-computed cache of prices for properties depending on various conditions so that they don't need to be computed each time on the fly and can be easily distributed to other services.

if the source (web) application makes their own audit tables. ex: our ruby on rails application uses the paper-trail gem

Start with https://github.com/paper-trail-gem/paper_trail

We use paper_trail for this

I think you could use that in conjunction with the paper_trail gem, as /u/GreenCalligrapher571 mentioned, which is also a good suggestion. As an additional note, when changing records in production while using the paper_trail gem, I suggest wrapping your database-mutating statements executed in the rails console within a whodunnit block, so PaperTrail.request(whodunnit: 'Dorian Marié') { widget.update name: 'Wibble' } or something rather than just widget.update name: 'Wibble'. Or, if you have some sort of issue-tracking / ticketing system, you could set the whodunnit value to the ticket number or whatever, and then anyone who wants to know why the records are in the state they're in can consult that ticket, which hopefully has additional relevant context.

For a while we did this using the paper-trail gem. This was a very simple way to add a few lines of code to keep track of all of the changes made to an ActiveRecord model. But it came with one drawback. Every change to the data had to be done through ActiveRecord. There are often times when this makes an app vulnerable to a race condition. I’ll use a contrived example so as not to share any real code from our client’s app.

In most designs, this would probably not be possible unless a table tracking extension is being used in an ORM. Even with change tracking enabled through extensions like paper trail or Django simple history, it can be tricky to restore deleted entities. Object tracking would need to have been enabled before it is needed to ensure the data is still around to be restored.

Maybe you are looking for the papertrail gem? https://github.com/paper-trail-gem/paper_trail