Speed up docker-build job
This MR adds few changes that are highly speeding up the test image building job.
What was done?
Dockerfile
refactorization
1. Dockerfile
was refactorized to make it as much cacheable for docker builds as it can be (look into Docker docummentation for a reference):
-
ENTRYPOINT ["/tests/bin/run"]
andWORKDIR /tests
were moved to top because the will be probably never changed. -
bin/prepare
script was replaced by fewDockerfile
lines:-
RUN
statement (splited to a few lines to be more readable) which is installing system packages.System packages are updated least often.
-
COPY ./Gemfile* ./
andRUN bundle install
which is installing ruby gems.Gems - in general - are updated more often than system packages, but leas often than source code.
-
COPY ./ ./
andRUN chmod +x ./bin/*
which are storing rest of the sources in docker image.This is the most important part of the repository and will be changed a lot - more often than gems list and much more often than system packages.
-
2. Docker build cache support
Since Docker 1.10 it is impossible to use docker build cache on different machines by doing a simple docker pull
of previous version of the image. Image layers don't store parent chain anymore.
Docker 1.11 introduced a solution for docker build cache sharing between different docker hosts. It may be done with a proper combination of docker save
and docker load
.
build-image
job was extended with this approach and - in combination with a Runner's caching mechanism - we can now make usage of the docker build cache.
Please read this blog post by Runnable for more information.
Results
Please look on these two builds:
- https://gitlab.com/gitlab-org/gitlab-qa/builds/3670130 - first build,
- https://gitlab.com/gitlab-org/gitlab-qa/builds/3670130 - second build, after adding changes that not avoid system package nor gems layers cache.
What differences can we see:
-
Steps 1-6 of docker build are get from cache. Only steps 7 and 8 are really built in this job:
$ ./ci/docker-helper build Sending build context to Docker daemon 220.2 kB Step 1 : FROM ruby:2.3 2.3: Pulling from library/ruby Digest: sha256:6da2268270168527a72962ef1a83fff5fbdf6c0c672e8f39c4b10e027ca1e858 Status: Downloaded newer image for ruby:2.3 ---> e0f509c5f9ee Step 2 : ENTRYPOINT /tests/bin/run ---> Using cache ---> d74db0487b18 Step 3 : WORKDIR /tests ---> Using cache ---> 0125616850dd Step 4 : RUN sed -i "s/httpredir.debian.org/ftp.us.debian.org/" /etc/apt/sources.list && apt-get update && apt-get install -y --force-yes libqt5webkit5-dev qt5-qmake qt5-default build-essential xvfb git && apt-get clean ---> Using cache ---> 2c1f71b8b450 Step 5 : COPY ./Gemfile* ./ ---> Using cache ---> cc34c8219bb9 Step 6 : RUN bundle install ---> Using cache ---> 93457d7bbc08 Step 7 : COPY ./ ./ ---> b5d197cf8552 Removing intermediate container 33ca1656aaca Step 8 : RUN chmod +x ./bin/* ---> Running in 2e2836d17405 ---> 0846716eb772 Removing intermediate container 2e2836d17405 Successfully built 0846716eb772
-
Thanks to this we've significantly shortened build time!
First build took
13 minutes 13 seconds
to build and push docker image. Second build needed for this only5 minutes 21 seconds
. It's 60% faster!
What can we do more
We could consider to execute the build-image
step only on the dedicated runner.
Currently builds are executed on shared runners (auto-scaled machines) and on the dedicated runner. What this means:
- builds executed on the dedicated runner can't re-use cache from shared runners,
- builds executed on shared runners can't re-use cache from the dedicated runner,
- builds executed on shared runners need to store their cache in shared S3 cache server - it's a ~600MB ZIP archive that needs to be downloaded and uploaded each time build is executed.
If we decide to use only the dedicated runner for the build-image
step then we will drop the need to download/upload the cache archive (very time consuming) and also we will always have the latest cache available.
/cc @grzesiek