fast_page
Large Hadron Migrator
| fast_page | Large Hadron Migrator | |
|---|---|---|
| 2 | 3 | |
| 292 | 1,818 | |
| 1.0% | 0.3% | |
| 2.9 | 0.0 | |
| 10 months ago | 8 months ago | |
| Ruby | Ruby | |
| Apache License 2.0 | BSD 3-clause "New" or "Revised" License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
fast_page
-
We switched to cursor-based pagination
There are ways to mitigate the (although not eliminate) the slowing down of offset/limit pagination in later pages. The technique is called a "deferred join" and it is most effective in MySQL. The basic idea is to paginate as little data as necessary, and then do a self-join to get the rest of the data for a single page.
You can read more about it here: https://aaronfrancis.com/2022/efficient-pagination-using-def... or here https://planetscale.com/blog/fastpage-faster-offset-paginati....
There are libraries for Laravel (https://github.com/hammerstonedev/fast-paginate) and Rails (https://github.com/planetscale/fast_page) as well!
Cursor based pagination is wonderful, but sometimes you're stuck with offset/limit for whatever reason. Might as well make it fast.
-
Faster offset pagination for Rails apps
In both Django and Rails, passing a QuerySet or ActiveRecord::Relation as an argument to a filter/where creates a subquery - in this case, https://github.com/planetscale/fast_page/blob/main/lib/fast_... creates SELECT ... WHERE id IN (SELECT id FROM ...).
So it's not exactly an inner join, but in theory it will be optimized to the same query plan as an inner join. That said, https://stackoverflow.com/questions/2577174/join-vs-sub-quer... has lots of commentary on why this shouldn't be depended on.
Large Hadron Migrator
-
GitHub downtime root cause analysis
No you didn't. They're doing what is often referred as "online schema change" using https://github.com/github/gh-ost (but the concept is the same than percona's pt-online-schema-change, or https://github.com/soundcloud/lhm.
-
Database... or Goose?
Is there anything similar for MySQL? There is https://github.com/soundcloud/lhm but it's pretty much outdated nowadays
-
Do you use migrations for data manipulations? What are the pro's and con's ?
I may do it from the console or a task if I wanted to modify a large number of records, e.g. something in my Users table. I think you need a sense of how long the update will take - I'm not sure if there's any issue with migrations timing out or such like. If I modify my Users schema it takes 5 minutes or so as it has to make a copy of the table and swap it in and that works fine - https://github.com/soundcloud/lhm
What are some alternatives?
fast-paginate - A fast implementation of offset/limit pagination for Laravel.
Squasher - Squasher - squash your old migrations in a single command
Scenic - Versioned database views for Rails
Lol DBA - lol_dba is a small package of rake tasks that scan your application models and displays a list of columns that probably should be indexed. Also, it can generate .sql migration scripts.
PgHero - A performance dashboard for Postgres
Foreigner - Adds foreign key helpers to migrations and correctly dumps foreign keys to schema.rb
Seed dump - Rails 4/5 task to dump your data to db/seeds.rb
BatchLoader - :zap: Powerful tool for avoiding N+1 DB or HTTP queries
DatabaseValidations - Database validations for ActiveRecord
Blazer - Business intelligence made simple
Seedbank - Seedbank gives your seed data a little structure. Create seeds for each environment, share seeds between environments and specify dependencies to load your seeds in order. All nicely integrated with simple rake tasks.