Redundancy fails when using registry.
Summary
On self hosted gitlab and runners, when having a shell runner with concurrency > 1 it fails when multiple docker registry login happen simultaneusly.
Steps to reproduce
Make 2 jobs happen at the same time.
- Job A logs in and writes token "X" on `$HOME/.docker/config.json`
- Job A starts building an image.
- Job B logs in and writes token "Y" on `$HOME/.docker/config.json`
- Job B starts building an image.
- Job A finishes building.
- Job A pushes to registry and gets `unauthorized: HTTP Basic: Access denied`
Actual behavior
Jobs fail.
Expected behavior
Runners should save the token somewhere else / share tokens / always should give them the same token so multiple jobs can interact with registry simultaneously.
Relevant logs and/or screenshots
running with gitlab-ci-multi-runner 9.5.0 (413da38)
on molniya.local (c90ad379)
Using Shell executor...
Running on 1bcc66369c10...
Cloning repository...
Cloning into '/home/gitlab_ci_multi_runner/data/builds/c90ad379/0/[private info]'...
Checking out 493639f7 as lk_validation...
Skipping Git submodules setup
$ git config --global user.name "Gitlab CI"; git config --global user.email "[private info]"; git merge --no-commit --no-ff origin/master
Automatic merge went well; stopped before committing as requested
$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN [private info]
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
$ git submodule update --init --recursive
$ docker build --pull -t ${DOCKER_IMAGE_NAME}_TEST -f dockerfiles/test/Dockerfile .
Sending build context to Docker daemon 10.08MB
Step 1/12 : FROM [private info]
amd64: Pulling from [private info]
01f86be7956d: Already exists
dea9351eecf4: Pulling fs layer
60ea68f7f96b: Pulling fs layer
8cbdb3c22e5d: Pulling fs layer
87005c9b4b0c: Pulling fs layer
3fcb6f2f21e7: Pulling fs layer
7ab8249f2b20: Pulling fs layer
f89df1b56609: Pulling fs layer
9fc21a876877: Pulling fs layer
03e641177b91: Pulling fs layer
0f3ca707795e: Pulling fs layer
bd2efd67d0c4: Pulling fs layer
24ee651d2322: Pulling fs layer
be71be5f3b1e: Pulling fs layer
65195b2ed1eb: Pulling fs layer
03e641177b91: Waiting
87005c9b4b0c: Waiting
3fcb6f2f21e7: Waiting
0f3ca707795e: Waiting
7ab8249f2b20: Waiting
bd2efd67d0c4: Waiting
f89df1b56609: Waiting
24ee651d2322: Waiting
9fc21a876877: Waiting
be71be5f3b1e: Waiting
65195b2ed1eb: Waiting
dea9351eecf4: Verifying Checksum
dea9351eecf4: Download complete
8cbdb3c22e5d: Verifying Checksum
8cbdb3c22e5d: Download complete
60ea68f7f96b: Verifying Checksum
60ea68f7f96b: Download complete
3fcb6f2f21e7: Verifying Checksum
3fcb6f2f21e7: Download complete
7ab8249f2b20: Verifying Checksum
7ab8249f2b20: Download complete
f89df1b56609: Download complete
03e641177b91: Download complete
dea9351eecf4: Pull complete
0f3ca707795e: Verifying Checksum
0f3ca707795e: Download complete
60ea68f7f96b: Pull complete
bd2efd67d0c4: Verifying Checksum
bd2efd67d0c4: Download complete
24ee651d2322: Download complete
be71be5f3b1e: Verifying Checksum
be71be5f3b1e: Download complete
8cbdb3c22e5d: Pull complete
65195b2ed1eb: Verifying Checksum
65195b2ed1eb: Download complete
87005c9b4b0c: Download complete
87005c9b4b0c: Pull complete
3fcb6f2f21e7: Pull complete
7ab8249f2b20: Pull complete
f89df1b56609: Pull complete
9fc21a876877: Verifying Checksum
9fc21a876877: Download complete
9fc21a876877: Pull complete
03e641177b91: Pull complete
0f3ca707795e: Pull complete
bd2efd67d0c4: Pull complete
24ee651d2322: Pull complete
be71be5f3b1e: Pull complete
65195b2ed1eb: Pull complete
Digest: sha256:78ab93673731ae5cee63d2747371e1fe9c296987408fc70818c937090e818913
Status: Downloaded newer image for [private info]
---> 5ee0f678a15c
... some work done
Step 12/12 : WORKDIR /usr/src/app
---> 67c9dbf160b5
Removing intermediate container 6c20c7375af7
Successfully built 67c9dbf160b5
Successfully tagged [private info]:lk_validation__TEST
$ docker push ${DOCKER_IMAGE_NAME}_TEST
The push refers to a repository [private info]
88f61d8a0986: Preparing
a92f3563f094: Preparing
a2f3951919a4: Preparing
d3c694b0625d: Preparing
072d28a4af76: Preparing
465c70e78607: Preparing
e4ef306bdfe6: Preparing
0688f17d3cee: Preparing
ebfcb0078351: Preparing
54765e6b2aec: Preparing
a73fe25502c6: Preparing
1992be944a38: Preparing
ee2326f56424: Preparing
706f8757ab3e: Preparing
0010619c5b8b: Preparing
7f294cade997: Preparing
9694784a321b: Preparing
bec67f112db3: Preparing
c8886e202cc5: Preparing
1e357753ddf8: Preparing
679051392a7e: Preparing
c5400bf64d6e: Preparing
4a9ded4bcac9: Preparing
e6480b4e39de: Preparing
ee2326f56424: Waiting
465c70e78607: Waiting
e4ef306bdfe6: Waiting
706f8757ab3e: Waiting
0688f17d3cee: Waiting
0010619c5b8b: Waiting
ebfcb0078351: Waiting
54765e6b2aec: Waiting
7f294cade997: Waiting
a73fe25502c6: Waiting
1992be944a38: Waiting
9694784a321b: Waiting
bec67f112db3: Waiting
4a9ded4bcac9: Waiting
c8886e202cc5: Waiting
e6480b4e39de: Waiting
1e357753ddf8: Waiting
c5400bf64d6e: Waiting
679051392a7e: Waiting
unauthorized: HTTP Basic: Access denied
ERROR: Job failed: exit status 1
Used GitLab Runner version
running with gitlab-ci-multi-runner 9.5.0 (413da38) Using Shell executor...
Related Issue
Edited by username-removed-782550