Skip to content

The `rake db:rollback` job can pass on CE but fail on EE

Fun fact that I discovered today while working on Wednesday's CE->EE merge (https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1427).

There was the following error in the rake db:rollback job:

StandardError: An error has occurred, all later migrations canceled:

Index name 'index_project_authorizations_on_project_id' on table 'project_authorizations' already exists/builds/gitlab-org/gitlab-ee/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/schema_statements.rb:954:in `add_index_options'
/builds/gitlab-org/gitlab-ee/config/initializers/mysql_ignore_postgresql_options.rb:44:in `add_index_options'
/builds/gitlab-org/gitlab-ee/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:540:in `add_index'
/builds/gitlab-org/gitlab-ee/config/initializers/mysql_ignore_postgresql_options.rb:23:in `add_index'
/builds/gitlab-org/gitlab-ee/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/migration.rb:665:in `block in method_missing'
/builds/gitlab-org/gitlab-ee/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/migration.rb:634:in `block in say_with_time'
/builds/gitlab-org/gitlab-ee/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/migration.rb:634:in `say_with_time'
/builds/gitlab-org/gitlab-ee/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/migration.rb:654:in `method_missing'
/builds/gitlab-org/gitlab-ee/lib/gitlab/database/migration_helpers.rb:26:in `add_concurrent_index'
/builds/gitlab-org/gitlab-ee/db/migrate/20170130204620_add_index_to_project_authorizations.rb:9:in `up'

The reason this fails is because the #down method for this migration does nothing on MySQL (because we'd drop the foreign key along with the index in that case, see f67d8eb1). This means that if you rollback this migration and redo it, an error is thrown because the index is already (== still) there!

That makes sense and the fix is to check if the index exists before attempting to create it.

But then I wondered, why didn't this fail in CE? That's because the job is rollbacking an arbitrary number of migrations (120). Remember that EE has additional migrations compared to CE (currently 541 migrations in CE, 645 in EE), so in CE the CreateProjectAuthorizations migration was rollbacked (as you can see in https://gitlab.com/gitlab-org/gitlab-ce/builds/12439000), thus when we try to migrate again, the table and the index are created successfully.

But on EE, the CreateProjectAuthorizations migration is more than 120 migrations before the latest migration, meaning that it's not rollbacked.

As a short-term solution, I've fixed the AddIndexToProjectAuthorizations migration to skip the index creation if it already exists, but as a long-term solution, we could rollback everything with bundle exec rake db:rollback STEP=10000. I guess this case was really a case of bad luck, though.

/cc @godfat