Persistent deployment keys/tokens to fetch registry images outside of GitLab

Added ~19173 feature proposal labels

Removed ~19173 label

Milestone changed to %Backlog

I would love to help with this. I'm currently working on using the Gitlab Container Registry to deploy to our team's kubernetes stack, and it's a bit clumsy to have a separate kubernetes user connected to gitlab. I would love for the ability to connect to the registry just by using a deployment key, similar to how the git deploy keys are managed.

If someone could point me in the right direction for helping with this, that would be appreciated.

Maybe we should extend deployment keys with deployment credentials concept that would allow you to access all project related data in read-only mode?

@JobV @markpundsack what do you think?

@ayufan makes sense to me. I'm sure @markpundsack is excited to work that out.

I am on board. I currently use specific images to build many projects in the same group and its terrible to have to replicate them across all repos I was going to suggest having an option to make a registry available for the entire group, but the deploy key seems better. :-)

@ayufan I think I'd have to see a full proposal to judge. Would this solve all the problems in https://gitlab.com/gitlab-org/gitlab-ce/issues/18994?

Perhabs this relates to #19275 (closed)? What I am currently missing is a clean way of using a private image from the registry as the base image of another ci build. Can the deploy keys help solving this?

@ttsiebzehntt Do you refer to image stored in Container Registry, or image stored somewhere else?

@ayufan stored in the Container Registry. There is a private Gitlab project A which builds a docker image and stores it into the Container Registry. Another Gitlab Project B wants to use this image as the base image for its CI build.

If built-in in GitLab Container Registry, so we are thinking how we can resolve that issue.

Another vote for this -- it would significantly ease our continuous deployment burden. We currently pull images automatically created by our continuous integration down manually, retag, and push to a different private registry. Having this would allow us to use the gitlab registry without requiring us to create, manage (and since we are an EE user, pay for) a deployment user.

Another +1 for this

Added ~480950 label

mentioned in issue gitlab-com/support-forum#923 (closed)

Mentioned in issue #17801 (closed)

As I've understood, the GitLab CI use case is resolved by the time-based token $CI_BUILD_TOKEN. But deploying the built image on a Kubernetes cluster, for example, still needs something like the "deploy key". Is this correct? Or have I missed something?

My use-case is using docker registry to allow our customers to pull our closed source app's docker images so they can install our app on-prem. With the current setup, each new customer would have to create a gitlab account and we would have to add them to the project for them to have pull permissions. I'd much rather if there was a way for us to generate deploy tokens for registry auth that we can control.

My current workaround is to set up my own docker registry because of this issue and this one: https://gitlab.com/gitlab-org/gitlab-ce/issues/17801

I set up the auth with SSH tunneling so they can use the same private key for authenticating against both gitlab to pull down some repo and my own docker registry. It mostly does the trick but it's really not very nice. I'd hope this would be a temporary workaround, fixing the auth problem would save me a good amount of trouble.

added ~113220 label

EE customer interested in this feature ZD #53410

I am EE customer too, who is interested in this feature. Is there a way to register official interest by EE customers?

@gajus you can send a support ticket to indicate interest for features.

added ~1071780 label

I would also love to see this so we can deploy directly from the container registry. At the moment we Nexus OSS as our docker repo for deployment because of its permissions. It would be awesome if we could some sort of read only deployment keys so we can deploy directly from the container repo :)

added shortlist label

Another vote for this from me. We've got several customers that use our images on premise that we need to be able to authenticate without giving them a full user on our registry.

added ~19492 label

@pedroms - Can you help out with this? Thanks!

Another vote for this! Setting repos to public is at the moment the only option to incorporate the registry in configuration management tools such as SaltStack.

+1 for this feature

Totally, as we move into CD and leveraging Docker with environments this is getting critical fast.

/cc @joshlambert

+1 would be really useful to deploy to kubernetes or docker swarm!

+1 we need it to deploy on kubernetes dev cluster

@ayufan Would you be able to write up a little more detail on your deploy account idea? Is this basically an account within GitLab which would be scoped per project (or potentially group) and have no rights other than RO access to container registry?

added ~19173 label

added ~897282 label

removed ~480950 label

added auto devops label

We may also need this for our auto deploy feature for private projects.

When working on the pipeline triggers API, we discussed a future iteration where you could specify which projects to provide access to, and a possible implementation might have involved fake "users" to model permissions based on. Perhaps this is a similar case.

changed title from Having deployments secret for registry to Persistent deployment keys/tokens to fetch registry images outside of GitLab

Note that some part of this problem space has been solved recently by CI_BUILD_TOKEN which is a temporary token used by GitLab runners. But since it's temporary, it can't reliably be used by k8s or other services to pull the image later (on scale or restart events, for example).

Exactly, currently I must create fake user as well, since restarts/rescale on kubernetes are not possible with temporary tokens. It's not only important for kubernetes ingegration but for any other deployment method as well (mesos, swarm, rancher, etc.). Could you prioritise this?

Hey guys, this is a massive deal and is literally the first thing you run into when attempting to set up an automated deployment from within Gitlab (Wait... what do you mean I can't pull my images from gitlab??!) which was a big-ticket feature a while back and got everyone excited.

Without this feature the Kubernetes integration is pretty useless (or not as streamlined as it should be at all), and it means you essentially have to set up another Docker repository just for your deployment, and keep the secrets in sync. Please make this work with the current Docker registry!

@bikebilly @markpundsack Does it make sense to think about MVP here? This seems to be an important issue.

@grzesiek I'd love to see a minimal proposal here. Some options that come up:

Explicit deployment token creation and management via web UI (as mentioned in this issue's description). Downside: you have to manually put the token into a project variable.
Extend CI_JOB_TOKEN to be persistent. Downside: automatic token can escape
Create explicit CI_REGISTRY_TOKEN which is persistent. Downside: automatic token can escape
Push the secrets directly to Kubernetes. (I'm not sure that's any more secure.)
Use some kind of kubernetes namespace-based permissions...

First option seems the most viable and is the most widely applicable.

Any ideas for making this more automated? Or for making CI_REGISTRY_TOKEN secure?

removed shortlist label

The ultimate goal is to allow kubernetes to pull (not push) given image indefinitely using the same token (until the token is revoked). Making CI_JOB_TOKEN persistent defeats its temporary purpose. Creating CI_REGISTRY_TOKEN won't handle revoking, and honestly we need only one persistent token, not one per-pipeline. There are many workflows that developers are using to deploy to kubernetes, so please don't force one with 4. or 5.

This leaves us with 1. and I think this is the best solution - you can already create personal access token, just please allow to allow to attach explicit permissions to them. For example this token would disallow anything except for pulling from registry of any project user has access to (pushing disallowed for security).

1 is the only viable option. Keeping a secret in sync with Kubernetes and gitlab manually is not an arduous task and is perfectly acceptable for this case. Just making per project deploy tokens work would be the MVP.

As i understand it the deploy tokens are just a token, not a username and password pair, so perhaps that might cause some complexity, perhaps having a hard-coded username (docker-deploy? Gitlab?) would suffice.

1 is really the only viable one. Lets keep in mind the Kubernetes is not the only use case for this and I'd hate for GitLab to lose its usefulness by coupling itself tightly with a single container orchestrator.

Hard coding the username of "gitlab" seems fine and sensible default although I can see some may want this to be customisable if they want to hide gitlab from external users maybe?

Option 1 is the way to go. Just make it possible to manage pull tokens through the API as well.

One potential caveat, though, is that this requires one unique token per project. At least for our use case we group several apps (projects) in a Kubernetes namespace, and it would simplify things a lot if we can use the same pull secret for all the apps in that namespace. In Gitlab, this would mean the same token can be used to pull from several project registries.

Two suggested solutions:

Link the tokens at the group or project level. A group level token can pull from all project registries in the group.
Link the token to the user and inherit the user's permissions. This could maybe be implemented by adding a new scope "registry_pull" to the personal access tokens.

I love the feedback we received after posting options @markpundsack came up with. Thank you!

Option 1 indeed makes sense to me. We can think about injecting this access token into predefined variables automatically later (it should be possible). What do you think about it @bikebilly @ayufan?

Option 1 seems the best and since we're not using kubernetes but a bespoke solution it'll fit the bill nicely.

This leaves us with 1. and I think this is the best solution - you can already create personal access token, just please allow to allow to attach explicit permissions to them.

Like how the PATs already have api and read_user scopes? What if the PATs would have a read_registry scope?

@sindrepm I think it would be more secure to link it to the group. Linking it to the user would give it access to completely unrelated stuff (which is one of my problems with the current implementation of CI_JOB_TOKEN).

Link the token to the user and inherit the user's permissions. This could maybe be implemented by adding a new scope "registry_pull" to the personal access tokens.

Ah, I've been ninja'd by @sindrepm. This is how I see it working.

What if the PATs would have a read_registry scope?

This is what seems like a good solution for me as well.

I'd personally prefer something under the repository->deploy keys section, not tied to a particular user and instead to a project. But extending PATs is the fastest and easiest way to get this implemented then i guess it's acceptable.

As a workaround over here we're using dedicated bot users to manage project access rights and tokens the easiest way already (quite easy through impersonation, of course as long as you're an admin so I can see how that could be limiting for some).

I like option 1: - Link the tokens at the group or project level. A group level token can pull from all project registries in the group.

It is significantly much more effort. It is quite big task as it requires a UX, ~Frontend and backend to implement that. We would have to add new views, implement a new authorization mechanism in backend.

Limiting personal access token to only pulling is a small task. Probably doable in a few days as most of implementation is already there.

@bikebilly This is a support scheduling request to prioritize this issue as well

changed milestone to %9.3

added Deliverable label

In my original list, I should have had an explicit option to extend personal access tokens (as opposed to project tokens). Project tokens would be for a single project, whereas personal access tokens would be scoped to a user, much like CI_JOB_TOKEN is. I know there are reasons both will need to exist eventually, but as @ayufan pointed out, extending PAT is likely much easier as it just introduces a new scope. For that matter, the existing read_user scope may even work for people if we just made sure it worked with the registry (which I assume it doesn't, otherwise we wouldn't be having this issue).

We have a vision for making scopes more granular; letting the creator specify which projects/groups should have which access. Assuming that is an extension of personal access tokens (or at least the underlying authentication system), then that feels like a good place to invest. I'm not sure what code, if any, is shared between deploy tokens, personal access tokens, and CI_JOB_TOKENs.

@ayufan Is our medium to long-term solution more like to be based on personal access tokens?

Personal acccess tokens feels like the wrong thing to me. Using personal access tokens is only a tiny step on from just using my username and password.

At a project level several people have access. I might not be the only user configuring deployments which means with PAT there would be a mixture of people's access tokens in production.

Now let's say a member of staff leaves the company. Their account is disabled and access revoked. Suddenly a deployment somewhere starts failing because they'd used their personal access token. And now instead of having a reliable deployment key other project members have to go check every production system to make sure they didn't use their personal access toke somewhere else.

I think that would be a pretty critical break in usefulness there.

@haswalt Agreed it's not sufficient, but it's a great first step to making something possible, whereas today some flows are impossible. We'll continue to iterate after the first iteration.

@markpundsack We are more likely looking into extending Deploy Keys to be Deploy Tokens. As it makes a lot of sense in regards to allowing you to download code, download registry images, push code and push images.

For example currently Deploy Keys can be assigned to multiple projects, the same could be done with Deploy Tokens, allowing us to have very granular permission system if needed.

@markpundsack @ayufan we should make a decision on this issue's direction soon if we want this to ship. Let's do a meeting perhaps?

@dimitrieh It is decided. For %9.3 we do minimal change, allow to generate personal access tokens that can only pull container images.

@bikebilly @ayufan the question here is if you see this feature as more of a project trigger token (connected to your personal account/taking ownership, thus limited to the project as well as person) or as a instance wide token, based in your personal settings (only limited to your person.. so token will work for any container registry you have access to)

@ayufan Is it worth splitting the issue into two issues? One with target 9.3 with the limitations you describe and one for the backlog which allows for specific project/group tokens?

@bikebilly that seems like the more useful choice to me as well So this would mean putting it in your personal settings.

Although in the description is stated:

and it's complicated to handle registry rights to limit deployment users to pull-only or to specific groups / projects.

So it would mean this issue should include the option to limit it per project as well.

This is where I would like to know what the difference exactly is going to be vs personal access tokens, which already exist.

The give access to the api, but not to registries? If so, this will be hard to communicate, because effectively we are creating another kind of PAT correct?

So it would mean this issue should include the option to limit it per project as well.

@dimitrieh No, I don't think that would be part of the first iteration.

assigned to @dimitrieh

updated issue description

changed the description

added ~480950 UX ready frontend and removed UX labels

i'll stay assigned as responsible UX (multiple assignees ftw)

@bikebilly would be okey with that! updated issue description with:

changed the description

It looks OK for me! Thanks

assigned to @zj

@bikebilly @ayufan How about making the scope read_registry to make it both clearer that it's read-only and to parallel read_user?

@ayufan @grzesiek Does read_user let you read from the registry as well? If not, isn't the MVC here to just make read_user read from the registry and not bother with a new scope at all?

Or, is read_user just user information and not any other information?

Does read_user let you read from the registry as well? If not, isn't the MVC here to just make read_user read from the registry and not bother with a new scope at all?

@markpundsack No, api scope is required. We use read_user scope just to make it possible to access users API.

Got it. So read_registry makes sense to me.

Yes, we should add read_registry.

@bikebilly description updated with:

changed the description

mentioned in issue #31740 (closed)

removed frontend label

mentioned in merge request !11845 (merged)

mentioned in issue #26074 (closed)

mentioned in commit 0b81b5ac

closed via merge request !11845 (merged)

This merged feature for 9.3 is really good... but I think there was a reasonable amount of agreement suggesting that "option 1" above was the best long-term goal (linking keys for deployment to projects/groups rather than users)

Is there a new ticket open to track any work towards this larger goal?

@bikebilly Can you create the follow-on issue(s)?

Umm... so, how do you use this token?

Documentations shows zero information on use. How to add this token into docker pull?

Is it like this?

docker login -u gitlab-ci-token -p TOKEN url
docker pull...

Asking for a friend.

Guys, could you please give a hint on what to do with kubectl after I've obtained a new Personal Access Token in my profile? I could not find anything on this in the GitLab docs, and the k8s ones are also confusing. It is suggested to call

kubectl create secret docker-registry regsecret --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

but this does not make sense, because there should be no need for username, email and password anymore. I'm expecting smth like

kubectl create secret docker-registry regsecret --docker-token=<your-token>

but this is obviously wrong.

Could you please help?

Just put any values in the other fields. I set the username to docker and the email to docker@docker.com.

Thanks for the hint @tom6! This worked for me:

REGISTRY_URL=registry.example.com ## my gitlab's registry
K8S_NAMESPACE=default
PERSONAL_ACCESS_TOKEN=?? ## from https://gitlab.example.com/profile/personal_access_tokens

kubectl create secret docker-registry regsecret \
  --docker-server=${REGISTRY_URL} \
  --docker-username=doesnotmatter \
  --docker-email=doesnotmatter@example.com \
  --docker-password=${PERSONAL_ACCESS_TOKEN} \
  --namespace=${K8S_NAMESPACE}

## to undo, just in case:
# kubectl delete secret docker-registry regsecret --namespace=${K8S_NAMESPACE}

Turns out that kubectl create secret docker-registry command needs to be repeated for every namespace, otherwise the generated secret is not visible. Also don't forget to mention regsecret in your deployments/pods:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: my-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
        version: v0.0.1
    spec:
      imagePullSecrets:
      - name: regsecret # this line is important!
      containers:
      - name: my-app-container
        image: registry.example.com/group/repo:master
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 100m
            memory: 20Mi
        ports:
        - containerPort: 80

Yay!

As far as I can tell on gitlab 9.4.2 this feature seems to be broken. I've got 2 PATs as an admin user with 2FA enabled.

When using the PAT with read_registry permission I get auth errors telling me permission denied. When I use the api permission I do have access.

@SamMousa @bikebilly Please create a new issue to track this.

mentioned in issue #35817 (closed)

As @sashk asked a month ago, it would be nice to have documentation on how to use this token. I'm trying right now and I don't see how I can make this work using docker login or docker pull.

@ayufan @markpundsack Can y'all help answer this question? https://gitlab.com/gitlab-org/gitlab-ce/issues/19219#note_38448358

Many people (including me) want to docker login using a deploy key but we don't know how to do it.

The solution is pretty simple in the end: docker login using the token as password. What you input as username doesn't matter. As I kinda stumbled into the solution, I'd rather have a dev do the doc than me.

mentioned in issue #37610

Thanks for the feedback. I created https://gitlab.com/gitlab-org/gitlab-ce/issues/37610 to track the documentation change.

mentioned in merge request !14785

Persistent deployment keys/tokens to fetch registry images outside of GitLab

Description including problem, use cases, benefits, and/or goals

Proposal

First iteration

Design

Links /references

Designs

Child items ...

Activity

Admin message

Admin message

Persistent deployment keys/tokens to fetch registry images outside of GitLab

Description including problem, use cases, benefits, and/or goals

Proposal

First iteration

Design

Links /references

Activity