Make the CI migration smoother

After some brainstorming @marin and I came up with the following ideas to make the CI migration smoother.

Main ideas:

• The final CI export should create just 1 archive containing a DB dump and the build logs
• It should be possible to import a 'final CI export' into a GitLab 8.0 instance while it is online
• We should have a final 7.14.x CI release that contains all the necessary scripts

Then the upgrade procedure would be:

On CI

• Upgrade CI to final 7.14
• Shut down CI, create backup
• Generate export archive using a single rake command

On GitLab

• Update GitLab to 8.0 and turn off its CI
• Copy the export archive to the GitLab server (which is online!)
• Import the CI export archive (with GitLab online)
• Enable CI in GitLab 8.0
• (omnibus) update CI to 8.0 to get the NGINX redirects

Doing it like this decouples the CI migration from the GitLab 8.0 update, and it avoids long combined downtime.

If we do this we need to improve the CI export/import scripts, make it possible to disable CI in GitLab 8.0, and release the final 7.14 CI version that has the integrated export scripts.

What do you think @ayufan ?

cc @marin @jnijhof @dzaporozhets

Use a migration in the final 7.14 to rename tables and indexes

Vendor the mysql-postgres converter in the final release. Admin runs rake final_export[postgresql] (where postgresql is the DB used by GitLab 8.0).

Instead of 'final CI 7.14' just make it CI 8.0.

Deal with accidental 7.x -> 8.1 omnibus updates omnibus-gitlab#794 (closed)

mentioned in issue #2565 (closed)

mentioned in issue #2566 (closed)

New plan:

Release CI 8.0, 'final migration edition'. It has DB migrations that prepare for the migration gitlab-org/gitlab-ci#314 and an easy to use export rake task gitlab-org/gitlab-ci#315.

In GitLab 8.0, we add a 'disable CI' option to allow online CI import #2565 (closed) and we adapt the existing CI import script to work with the improved CI export script #2566 (closed).

@marin can ensure that the migration gets deployed to GitLab.com on Monday during the European workday? I hope that @jacobvosmaer and @ayufan will help you to get it finished and deployed as early as possible. We need to release 8.0 on Tuesday at 12am CET so that only leaves time on Tuesday morning to fix any critical bugs that surfaced after the migration. To give these bugs time to surface I think it is essential we deploy and RC that includes the migrations on Monday as soon as possible. /cc @rspeicher

GitLab CE part: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1365

Migration guide: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1335

To test migration you need to fetch sources:

cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H git fetch origin
sudo -u gitlab_ci -H git checkout origin/export-script
sudo -u gitlab_ci -H bundle install
cd /home/git/gitlab
sudo -u git -H git fetch origin
sudo -u git -H git checkout origin/optimize-ci-tags-migration
sudo -u git -H bundle install

And follow https://gitlab.com/gitlab-org/gitlab-ce/blob/d511edc952574d0e5f51180648cbfbd2061bfe47/doc/migrate_ci_to_ce/README.md

rspeicher: Updated link.

@rspeicher @sytses RC3 was deployed and ci.gitlab.com migrated to GitLab.com/ci

@jacobvosmaer can this be closed?

@sytses we are not done with this topic but I think we do not need this particular issue anymore.

Status changed to closed

mentioned in merge request !11936 (merged)

mentioned in merge request !14785