Skip to content

Allow not blocking startup process when Gitlab is reconfiguring

James Nutt requested to merge jmd-skip-waiting-for-reconfigure into master

What does this MR do and why?

When starting up GitalyCluster which is a we need to start multiple gitlab/gitlab-ee docker containers.
These include 1 * praefect, 3 * gitaly, 1 * gitlab

In our current implementation, we run docker run -d ... and then docker attach ... and wait until we find a matching gitlab reconfigured string to determine the service is ready.
As this blocks, it takes several minutes for each service to be ready, before moving to the next. That in turns means it can be quite slow to get the entire environment read.

Take for example a recent job https://gitlab.com/gitlab-org/gitlab-qa/-/jobs/2393025617 where it took from 00:28:06 -> 00:36:11 -- 8min5sec to start the environment

[date=2022-04-29 00:28:06 from=Gitlab QA] INFO  -- Docker shell command: `docker run -d -p 80 --name gitaly1
[date=2022-04-29 00:29:13 from=Gitlab QA] INFO  -- Docker shell command: `docker run -d -p 80 --name gitaly2
[date=2022-04-29 00:29:57 from=Gitlab QA] INFO  -- Docker shell command: `docker run -d -p 80 --name gitaly3
[date=2022-04-29 00:30:46 from=Gitlab QA] INFO  -- Docker shell command: `docker run -d --name postgres
[date=2022-04-29 00:30:55 from=Gitlab QA] INFO  -- Docker shell command: `docker run -d -p 80 --name praefect
[date=2022-04-29 00:31:49 from=Gitlab QA] INFO  -- Docker shell command: `docker run -d -p 80 --name gitlab-gitaly-cluster
....
....
....
gitlab Reconfigured!
[date=2022-04-29 00:35:59 from=Gitlab QA] INFO  -- Docker shell command: `docker port gitlab-gitaly-cluster 80/tcp`
0.0.0.0:49157
Waiting for GitLab at `http://docker:49157/help` to become available 200 -> GitLab is available.
[date=2022-04-29 00:36:11 from=Gitlab QA] INFO  -- Running Gitaly Cluster specs!
[date=2022-04-29 00:36:11 from=Gitlab QA] INFO  -- Docker shell command: `docker pull gitlab/gitlab-ce-qa:nightly`
nightly: Pulling from gitlab/gitlab-ce-qa

As we can see from the timings, gitlab-gitaly-cluster is the major blocker here, taking 4m10s to complete, whereas the other services are in the 1min-2min range.
By running these other services in the background, we can save several minutes of startup time

Sample improvement

Using a job from this MR we can see the startup time is reduced from 8m5s to 4m43s.

[date=2022-05-02 01:51:10 from=Gitlab QA] INFO  -- Docker shell command: `docker run -d -p 80 --name gitaly1
...
...
[date=2022-05-02 01:55:53 from=Gitlab QA] INFO  -- Docker shell command: `docker port gitlab-gitaly-cluster 80/tcp`
0.0.0.0:49157
Waiting for GitLab at `http://docker:49157/help` to become available 200 -> GitLab is available.
[date=2022-05-02 01:55:53 from=Gitlab QA] INFO  -- Running Gitaly Cluster specs!


Total setup time: 4:43

How to set up and validate locally

  1. Run E2E tests using gitlab-qa as normal. Note that GitalyCluster & Praefect Scenarios should have a quicker startup time.
e.g. 

./exe/gitlab-qa Test::Integration::Praefect EE --no-tests --no-teardown
./exe/gitlab-qa Test::Integration::GitalyCluster EE -- qa/specs/features/api/3_create/gitaly/praefect_repo_sync_spec.rb --tag orchestrated

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by James Nutt

Merge request reports