GitLab container registry

The GitLab Runner is be able to build a container image

It will also soon be able to upload build artifacts.

If it uploads a container image we should make that available via the docker image interface so that you can use GitLab as a private container registry, an alternative to Docker Trusted Registry.

Instead of using the upload build artifacts function it might be easier to use native Docker upload functionality docker push my-registry:5000/my-image.

/cc @ayufan @dzaporozhets @DouweM @JobV

mentioned in issue #3286 (closed)

Added ~19173 label

There is also an open source product from Docker that doesn't have a web interface, only an API https://github.com/docker/distribution and it is missing some functionality, such as deleting old images.

@ayufan your thoughts on this?

It would be great to introduce this with 8.6 in some form (earlier if it's just kamil building it).

@JobV I do have some ideas about the whole Deploy and Container Registry. Will post that after we finish the current release cycle.

@JobV If we decide that we want this we could compile it in GitLab-Workhorse someday :P This what others do :)

You guys might have a look at Portus, an open-source Docker Registry authentication service and web UI (also a rails app). You pair this up with an official registry:2 and you basically have an open-source Docker Hub. This is what I'm in the process of setting up for my current project.

Thanks @jonathon-reinhart

@dzaporozhets @DouweM @JobV @stanhu @ayufan I believe that a docker registry is essential for a modern CI/CD workflow. We should ship one with GitLab for our on-premises customers.

@JobV please add to direction and schedule for 8.7?

@sytses I agree. It will make it easy to integrate with our planned workflow.

Added ~122770 label

@ayufan this is doable for 8.7?

@sytses I'm a long time supporter of this

Milestone changed to 8.7

@JobV

This requires:

Track 1:

1. Building plugin system
2. Building plugins that can use our registry

Track 2:

1. Add container registry as module to GitLab (another moving part...)
2. Built authentication system integrated with GitLab
3. Expose the data to the builds

(The two can be done in parallel)

I expect that the first realistic time is for this is at least 8.8 (if we manage to do everything as planned, which is not obvious :()

We (@fh1ch @bachp @bufferoverflow) really like that idea and discussed this internally as well. The solution we had in mind is the following:

Use GitLab as an Authorization Service according to Docker distribution auth specification and configure a decoupled Docker Registry using this service, that can be shipped easily via omnibus.

So the main work from our perspective is:

• add a JSON Web Token API endpoint to GitLab, e.g. /api/jwt
• map group/repo concept and permissions from GitLab 1:1 to Docker, e.g. gitlab-org/gitlab-ce (git repo equals Docker Image name)
• map the GitLab permissions roles used within GitLab to Docker Registry permissions, e.g:
  • owner/master => push, pull
  • developer/reporter => pull
  • public/internal repos => pull

Here is a sample of a Docker JSON Web Token, reflecting this:

{
    "iss": "gitlab.com",
    "sub": "bufferoverflow",
    "aud": "registry.gitlab.com",
    "exp": 1234567890,
    "nbf": 1234567890,
    "iat": 1234567890,
    "jti": "tYJCO1c6cnyy7kAn0c7rKPgbV1H1bFws",
    "access": [
        {
            "type": "repository",
            "name": "gitlab-org/gitlab-ce",
            "actions": [
                "pull"
            ]
        }
    ]
}

The ruby jwt gem also used within PORTUS does all the basics required for JSON Web Token.

@ayufan how about adding the registry in 8.7 and supporting it through a plugin in 8.8?

@bufferoverflow thank you for the suggestion!

I'm currently implementing wrapper for docker registry authentication as gitlab oauth2 application similar to what @bufferoverflow suggested.
I'm missing API call that would tell what access rights user has to a project. I know that user can see the project but not why. Owner is easy to detect others require more API calls (is namespace user or group, project members, group members, use highest access level) to determine push rights.

@sytses

• it should be quite easy to add,
• it requires to check how we should implement authentication,
• it would make sense to present a list of images in GitLab interface,
• we need to make sure that this will work on shared storage (not sure about that),
• I like the concept of 1:1 repository mapping, but is it always true that one repository is exactly one docker image?
• we need to think how to reliably delete docker images when project is deleted (we are introducing a separate component not connected with Rails, using external API) (this is important from security perspective),
• we need to think how will it be exposed to outside world: should it use separate vhost or should it use separate IP/or/port?
• whether we should use Docker Distribution or implement API by ourselves: https://docs.docker.com/registry/spec/api/ (the API is quite simple)

The amount of work required to have it working seems to not be that big, but there are a few design decision that needs to be resolved first.

I think strong decoupling is important and I suggest to use the docker api instead of implement it again and let GitLab be:

• the Authorization Service
  • other apps(e.g. npm registry) might use JSON Web Token as well in the future
  • a dedicated thing doing this simplifies testing and reuse
• a Docker Registry client to
  • push images via GitLab CI e.g.
    • for tags such as gitlab-org/gitlab-ce:v8.7.1
    • if configured via UI for each build gitlab-org/gitlab-ce:BUILDNR? => lot of storage...maybe non-sense, but great for debugging things within Eclipse Che or similar thing(#12759 (closed)) at a later step
  • display Docker images within UI
  • manage Docker images within UI
    • delete images
    • tag images (e.g. gitlab-org/gitlab-ce:v8.7.1 = gitlab-org/gitlab-ce:stable)

The Docker registry shall be fully independent from a distribution point of view and be exposed via dedicated DNS and IP.

I guess many apps might use it without direct relation to GitLab and there will be more consumers than producers.

Some people might use other apps then GitLab CI to build Docker images and might love to use the Docker Registry that uses the GitLab based JSON Web Token Authorization Service for the registry.

@bufferoverflow I like your ideas. Maybe we should start with evaulating the Authorization Service, what we need to make it work :)

@bufferoverflow I like your idea's too.

@ayufan "I like the concept of 1:1 repository mapping, but is it always true that one repository is exactly one docker image?" => we'll maybe need multiple docker images per repository because the development and production ones are different. One interesting idea is to make add the development code in later stages of the Dockerfile so you can use a previous layer of the same docker image for production

@sytses This is out of scope of Dockerfile.

I think that we can start with simple integration: integrated built-in Docker Registry, with 1-1 mapping, this will hold true for fairly long time. We can think later how to allow multiple images.

@bufferoverflow Excellent! Strong support for this! @sytses @ayufan This is not even a dockerfile problem. How you solve that is secondary. There is one point about fetching and pulling though, that the registry is able to content address and match layers if they are in the same image name space and differ in tags only. so something like: docker push registry/app:development and a posterior docker push registry/app:production would be ridiculously fast.

In any case, I strongly support flexible naming (it's not only about the image, it's about the tags!), so 1:1 mapping seems oversimplified.

The complicated part is repository vs registry vs image vs tag mapping. I don't believe it will be useful to integrate it into GitLab there is just too many use-cases and 1:1 mapping is just one very limited use-case.

@ayufan I made crude tool for authenticating via GitLab Oauth or private-token (fallback) and 1:1 mapping. No changes to GitLab were necessary. I used xanzy/go-gitlab, oauth2 and docker/distribution.

Mine last problem is how to integrate robot accounts without cluttering GitLab Users with robots.

@bufferoverflow I will make opensource registry authentication/authorization service for GitLab. But I need input on possible ACL use-cases and how to implement them. Our workflows are quite similar.

@mishak87 Do you share your WIP(work in progress) code somewhere?

I have some doubts if a dedicated service written in go is the way to go here, especially for a JSON Web Token based Authorization Service, which might be useful for a lot of other things beside of the the Docker use-case. JSON Web Token is the brother of OAuth and we do not now yet how they develop in the future. On the other side I see api scope and many other things in that area. I still believe that OAuth and JSON Web Token shall be brothers with the same mother(permission logic).

As far as I understand the architecture of GitLab currently(sorry, no deep dive until now), all the api things are handled via ruby within the GitLab core. How does the go app you are writing communicate with the core? Via OAuth api as far as I see, right?

1:1 mapping is a quick win and helps people to think about repo and naming structure. Of course there will be additional use-cases for other Docker Image Names, but as long as you are within your GitLab instance and the related Docker Registry, they should not collide with the repo namespace, otherwise many users might get confused on how things were built and deployed. If you want to push to gitlab/gitlab:stable , just create the group gitlab (or remove it as it is inactive anyway) and the master/owner will be able to push. Hmmm, might be worth to add a GitLab CI permission level as well.

A much more important thing I see is teams, e.g define teams that belong to several groups.

@ayufan I'm OK to start with a 1:1 mapping since we're not sure yet if we solve dev/prod difference through mapping, tagging, layers, or another way.

On 1:1 mapping features:

• option to allow only specific CI runners to push images
• SEMVER agnostic (tag latest, v0, v0.1, v0.1.0 from v0.1.0 git tag if it is the latest)
• integration for platforms that pull images ie.: Rancher - could be circumvented by adding (fake)users for each environment.

@bufferoverflow Yes to authenticate accounts I use oauth because project access level is provided in API. I have not implemented anonymous yet since I don't need them and they require different approach. There is no point in implementing JSON Web Token in Gitlab because it has only one scope api (read root access) and docker registry needs push and pull. So mapping must be done and that requires having an opinion.

I will be working on opensource version this week but first have to extend xanzy/go-gitlab API.

@sytses Basic principle of docker is that you use identical images for dev/prod. If you need something special for dev you should add it as a feature to the image. But this is out of topic. If you want help with that I would suggest asking on #docker IRC channel there is a lot of people that could help.

@mishak87 good to hear that it is best practise to use identical images. I wonder how this works for Rails apps, nice it is best practice not to bundle your testing gems for your production environment. Maybe dev/prod use different layers of the docker container? First do production install and then rebundle the gems for development?

@ayufan will this launch on the 22nd

@sytses Not sure yet. I just started on working on this.

@ayufan OK, thanks for the feedback

We've came up with a first proposal of implementation https://gitlab.com/siemens/gitlab-ce/commit/9fca07b8cbdb0ffecacd72d3dbf2370eff5c1fed for the Docker registry authorization part. It still needs some love and more tweaking (it's still a rough draft) but might become a MR anytime soon.

The implementation is similar to the proposal by @bufferoverflow, but utilizes the group visibility of GitLab for determining the access level and maps them to namespaces (like Docker-Hub).

mentioned in merge request !3738 (closed)

@ayufan @marin can we ship an image registry in 8.7?

@sytses I am not confident that we will ship it. On the package side omnibus-gitlab#1218 (closed) I am running into issues while testing, basically it has been very unstable for now.

MR by @ayufan gitlab-org/gitlab-ce!3787 is still WIP but I will let Kamil answer about how confident he is about that part of code being ready.

@marin thanks!

Milestone changed to 8.8

@marin this got bumped to 8.8 by Kamil

@sytses Thanks for the ping, talked with Kamil about it before the decision. I will shortly update the referenced issues and MR's.

Reassigned to @marin

This is finished.

Status changed to closed

mentioned in merge request gitlab-com/www-gitlab-com!4742 (merged)