Skip to content

WIP: POC for zero downtime migrations without having to manually manage what migration goes into what release

Douwe Maan requested to merge release-migrations into master

@yorickpeterse @rspeicher @smcgivern @stanhu I'd like to get your thoughts on this.

This is a POC for zero downtime migrations where the developer and release managers don't have to manually manage what migration goes into what release, everything can be merged at one time in one MR, and GitLab itself is instead responsible for performing the right migrations and the right code level changes (like automatically writing to the new column, or ignoring the old column) at the right times.

This will potentially make developer and RMs lives a lot easier, while reaching the same result as described in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9976.

This POC just covers renaming a column, but can easily be adapted to do the same for other kinds of changes, by adding a ChangeableColumnType module based on RenameableColumn, for example, with migrations to match.

The example renames users.username to users.handle in 3 steps, that are only run by rake db:migrate when VERSION holds the right release:

  • 9.0.0 - Add handle column
  • 9.0.0 - (post deploy) Migrate data from username to handle
  • 9.0.1 - Remove username column

You can try it out by playing with the version number in VERSION, running rake db:migrate, and then running rails console. You'll see instructions for what to do with the code logged as info to log/development.log or as a warning to STDOUT after the final migration has run, and you'll be able to see that the right ignores, aliases and automatic writes to the new column will be set up depending on what migrations have run. Use rake db:rollback STEP=n if you get scared and want to take n steps back.

A downside is that since these future migrations are not actually run locally or on CI, db/schema.rb isn't updated, and there's no way of knowing for sure if code will really continue to work once the migrations "magically" run on production. Which is of course a bad thing. Developers can run these migrations locally by toying with VERSION, but we can't be sure that they actually did this, and we can't expect them to run all specs locally.

To resolve this, we could only move migrations from db/release_migrations/[version] to db/migrate or db/post_migrate at release time (by the release-tools) in a commit that also updates db/schema.rb and subsequently gets its pipeline run. There would not be any extra code changes required at that point if the developer followed the instructions that RenameableColumn outputs, since RenameableColumn automatically makes appropriate runtime changes by looking at what migrations have run.

To make this easier to use for developers, we can add a generator that automatically generates the 3 migration files in the right places, adds the rename_column call to User, and again prints instructions on what code changes (not) to make.

It may also be worth considering using an unreleased folder like we do for changelogs, so that the user doesn't need to predict what release the MR will go into. Note that it's fine if the there are migrations for 9.0.1, and 9.0.1 is never released, since migrations are run when the current release >= the specified release. This means that these migrations would be run in 9.1.0, which is still after 9.0.0, as intended. It's NOT fine if the migrations are in 9.0.0 and 9.0.1 folders, and the MR as a whole ends up being released in 9.0.1 rather than 9.0.0, since then all migrations will happen at the same time, instead of with the intended wait of one release.

Merge request reports