Intended use of docker registry in `.gitlab-ci.yml` with private repository for project

Your runner needs to be able to access your gitlab registry. You can create a user account on your gitlab, add the user as a reporter to the project containing the image, and then with the user that's running gitlab-ci-multi-runner on your server execute a docker login <your registry url>. Be aware that this will store the password in essentially plaintext (base64 encoded) in ~/.docker/config.json. Once that file exists, the runner will be able to access the registry.

Added ~19173 container registry labels

Thanks for an answer. I still don't fully understand. So I would need to perform the login on the server, right? Does the name of the user have to be the same on the server and for the user in my project? Can you elaborate more please? Also which password will be stored in the config? Do I need to store it or will it be stored by gitlab?

I would expect this to function approximately as if I would be using Docker Hub under image: setting (e.g. ubuntu:14.04). I do not necessarily need to modify the image in each build, I would just need to have the read-only access without creating any users. I would really like a very simple setup, so that I could send a simple tutorial to the users on how to set this up and avoid answering hundreds of questions when something goes wrong. :)

Yes, you need to log in on the server. The username doesn't have to be the same, but you have to log in with the user on your server that is running gitlab-ci-multi-runner (if you installed it from the debian package and followed the default instructions, it'll be running as root). The password (and username) that is stored is the one you give to docker login gitlab.ipb.uni-bonn.de:4567. I recommend you create a specific account for this, and only give it reporter access to the images it needs to pull. docker login will create the file ~/.docker/config.json, store the username and password there, and then when the runner tries to pull the image, it'll read the username and password from there.

If you want to use the image in your .gitlab-ci.yml, the runner needs to be able to pull it. For docker hub images that's not a problem because they're public. For your own gitlab server it'll work if the project is public (but see also #19117 (closed)), but if you want to use images in private projects, you need to log in the runner and give its user access.

You only need to do this docker login once for each server/gitlab-ci-multi-runner installation. Once the ~/.docker/config.json file exists, the runner will be able to authenticate.

Ok, thanks. So basically, I need to create a dummy user, give him reporter access to the repo, login this user on the server (with sudo, because it's installed from debian package) and store this users name and password in ~/.docker/condif.json (is this in home directory on the server?) and then everything will be working? Correct me if I am wrong anywhere please.

docker login will store the username and password for you, you just need to run sudo docker login gitlab.ipb.uni-bonn.de:4567 and then type in the username and password.

Ok, thanks. Now it's clear. Sorry for taking so long to understand.

However, I think this should also be somewhere in documentation. Or did I just miss it somewhere? If it is not there I may do a pull request for documenting it tomorrow after I will try it out.

@zabugr I can't find a reference to it, but it was said clearly that currently GitLab Runner doesn't support private container registry for images.

@ayufan this is exactly the point. I believe it should stand out better. To me it was a bit confusing and I haven't seen it in the documentation.

Yeah, my solution is really just a workaround.

It still doesn't work for me.

I have logged in with the credentials of a user that is a reporter in my project on a server. ~/.docker/config.json file contents:

{
	"auths": {
		"gitlab.ipb.uni-bonn.de:4567": {
			"auth": "some_base64_number"
		}
	}
}

The error did not change. Still as reported in the issue.

You might need to restart the runner. This works for me, but I'm not running the runner as root and I use supervisor.

I did sudo gitlab-runner stop and sudo gitlab-runner start after that. Still the same result when I rerun old builds. @nkovacs, what do you mean by supervisor?

http://supervisord.org/

gitlab-ci-multi-runner runs as a user called cirunner, not root, and ~cirunner/.docker/config.json contains the username and password.

Hm, thanks I may give it a try. However, I strongly believe there should be a simpler way to use docker images from private registry.

UPD: I can pull the image from the registry being logged in to my server, where I am logged in as a proper user in docker. If I log out of docker, then the error is the same that I see in ci.

UPD2: For now it is fine for me that I can manually download the image on the server and it is then available for my build (no updates though). But the original issue is not fixed.

Milestone changed to %Backlog

Changed title: Intended use of docker registry in .gitlab-ci.yml → Intended use of docker registry in .gitlab-ci.yml with private repository for project

mentioned in issue #19219 (closed)

I am also having this problem. I created an issue in EE https://gitlab.com/gitlab-org/gitlab-ee/issues/797.

mentioned in issue gitlab-com/support-forum#923 (closed)

Definitely need a way to do this on GitLab.com. I spent a bunch of time figuring out how to prepare a Docker image with all necessary dependencies (admittedly, it was only a weekend), only to figure out I couldn't actually use it to run the CI work I really wanted because it can't login.

@QuLogic : The only way I found so far to get it working in gitlab.com is doing it the plain old docker way:

 image: docker:latest
services:
  - docker:find
before_script:
  - docker login -u <user> -p <password> registry.gitlab.com
  - docker run -d --name <my_service> registry.gitlab.com/<group>/<build_name>
test:
  - docker build -t <current_project_build_tag> .
  - docker run --link <my_service> <current_project_build_tag>

(see also https://gitlab.com/gitlab-com/support-forum/issues/923)

So, if I understand correctly there's no current way to fetch a private image in a runner, right?

What I'm trying to do, (using a python example):

variables:
  # The Gitlab Registry
  GL: "registry.gitlab.com"
  # Docker Hub Image Name
  IMG: "username/project"
  # Experimental: test the overlayfs docker driver.
  DOCKER_DRIVER: "overlay"


stages:
  - build
  - test


build:python_image_build:
  image: "docker:latest"
  before_script:
    - "pwd"
    - "docker info"
    - "docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $GL"
  after_script:
    - "docker images"
    - "docker logout $GL"
  services:
    - docker:dind
  stage: build
  script:
    - "docker build -f Dockerfile -t $GL/$IMG:latest --pull ."
    - "docker push $GL/$IMG:latest"
  tags:
    - docker


test:run_tests:
  image: $GL/$IMG:latest
  stage: test
  script:
    - "python -m tests"
  tags:
    - "docker"

mentioned in issue #19865 (moved)

@rolandog currently not. As I mentioned in #19865 (moved), you can work around it by using the docker image and running the commands needed using docker API.

Similar to what another user posted, I set-up my registry with a new image through the night only to find this out after I was unable to leverage it in my yml file. This seems like a fairly important feature around the idea of centralizing your docker registry with the CI/CD process.

(FYI) In addition, as a follow-up to another user posting that this should be made clearer (this ticket or another one on the same issue), there was more than one social marketing post (e.g. blogs) that mentioned this concept (as value added) of being able to leverage the registry for private and proprietary instances.

This issue is in backlog unfortunately. But as there is more and more people having this issue I do hope anybody from gitlab will consider moving it up on the priority list.

Yes, It would be great if I could use more than one private registry with my runner.

Now I have to login back and for every time the runner execute stage in my pipeline.

Another option would to have ability to store more than one authentication token in the .docker/config.json for registry.gitlab.com or be able to store authentication token in the runner config.toml file.

Would it be possible to consider this issue before the next month's release? It seems pretty topical, according to the number of comments and likes.

Ping @ayufan (see prev comment). Could you please pass this idea to other GitLab devs who could be potentially interested in taking this?

As far as I'm aware, there's special user named gitlab-ci-token, and it has permission to access image in Gitlab Container Registry, as described in runner documentation: Using the GitLab Container Registry section of https://docs.gitlab.com/ce/ci/docker/using_docker_build.html . Unfortunately I found no way to provide these credentials in a way allowing to pull image from the "image" property inside .gitlab-ci.yml .

It seems natural for me to use image from Gitlab Registry with Gitlab CI, they are parts of the same service after all, so I'm really looking forward to that.

Mentioned in issue #24024 (moved)

@alexander.kachkaev.wb @mikz1

We plan to solve a the GitLab Container Registry access in this release.

In next we will probably add some generic way for other registries.

@ayufan do you have an issue or an MR for interested parties to follow? And which is "this release" and "next release"?

@nikola1

We don't yet have MR. Next release is 8.14, so it will be released at 22.11.2016.

To make sure that there's no confusion what is shipped:

It will enable to use private docker images with image: specification from GitLab's Container Registry.

Here's the issue: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1828

Im my case the user running the gitlab-ci-multi-runner process (gitlab-runner) was not member of the docker group. (If I ran docker images, even without private registry, it returned Cannot connect to the Docker daemon)

After adding gitlab-runner to the docker group (usermod -aG docker gitlab-runner) the procedure with the secret variable in the project as described here https://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry worked

According to the New CI build permissions model, there should now be a token passed to the CI runner that enables this. At least for me, I am able to use an image from a private repo using just image: and no extra docker login when in gitlab.com.

If you're running your own instance, according to that page, you need at least GitLab Runner 1.8 to get the token parsing on the backend.

Yes, this works as expected now.

closed

Intended use of docker registry in `.gitlab-ci.yml` with private repository for project

Designs

Child items ...

Activity

Admin message

Admin message

Intended use of docker registry in `.gitlab-ci.yml` with private repository for project

Activity