unauthorized: authentication required on docker push to a different repo

I'm creating two docker images via gitlab-ci from one repository upon pushing them to GitLabs private container registry. The .gitlab-ci.yml is below. myproject is the group name. backend and docs are GitLab projects within this group. Both have the container registry enabled. build_image works OK but build_apidoc_image fails when pushing the image for the apidoc container to the registry with:

docker push myregistry.com/myproject/docs:latest
The push refers to a repository [myregistry.com/myproject/docs]
630816f32edb: Preparing
530d5553aec8: Preparing
c3f640b7175a: Preparing
378f94990a4e: Preparing
77b53f4affd3: Preparing
4b0bab9ff599: Preparing
d1c800db26c7: Preparing
42755cf4ee95: Preparing
4b0bab9ff599: Waiting
d1c800db26c7: Waiting
42755cf4ee95: Waiting
unauthorized: authentication required
ERROR: Build failed: exit code 1

The strange thing is that docker login succeeds:

$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN myregistry.com
Login Succeeded
$ docker build -t myregistry.com/myproject/docs -f apidoc/Dockerfile .
Sending build context to Docker daemon 485.4 kB

This gitlab build is triggered from the backend repository. When building and pushing manually to docs it works.

Here's the .gitlab-ci.yml

stages:
  - build

build_image:
  stage: build
  image: docker:git
  services:
    - docker:dind
  script:
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN myregistry.com
    - docker build -t myregistry.com/myproject/backend .
    - docker push myregistry.com/myproject/backend:latest
  only:
    - master

build_apidoc_image:
  stage: build
  image: docker:git
  services:
    - docker:dind
  script:
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN myregistry.com
    - docker build -t myregistry.com/myproject/docs -f apidoc/Dockerfile .
    - docker push myregistry.com/myproject/docs:latest

Added container registry label

I have the same problem on gitlab.com. I tried to login and push from my pc using:

docker login -u <my username> -p <my password> registry.gitlab.com
docker push registry.gitlab.com/<my group>/<my project>

but got the same error.

unauthorized: authentication required

Same problem here. I've a build script that pushes my docker image to gitlab registry and it worked fine for weeks. Since today it's not working anymore. I didn't make any changes on that script.

Same happens here since some minutes (;

We're looking into it. We did upgrade to 8.10.1 to fix some issues with strict host/port headers, but if this issue was present 2 days ago then there may be something else going on.

/cc: @ahanselka

mentioned in issue gitlab-com/infrastructure#256 (closed)

@stanhu For me this start after upgrade to 1.10.1.

@rodrigo.avila Right, there is a fix here: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/891. You can also use the workaround describe here. I'm investigating why the registry is behaving in this way.

1.10.2 fixes the issue for me. Does anyone else experience it or may I close this bug?

Status changed to closed

I'm seeing this very same problem while pushing by hand.

My guess is that it is related to communications timeouts or similar, as I'm pushing from a relatively slow DSL line. All except three of the layers went in, but the remaining ones fail once and again with this error or silently. See the failure, it finishes always the same, after a fair number of retries:

$ docker push <server>/<group>/<project>:<branch> 
The push refers to a repository [<server>/<group>/<project>]
a2827ad95b08: Layer already exists 
331769080c41: Layer already exists 
6663ba3560a9: Layer already exists 
5f70bf18a086: Layer already exists 
a69416df92de: Layer already exists 
5303c4fe3aea: Layer already exists 
614ab43b8532: Pushing [==================================================>] 120.6 MB
f35b3a860c21: Layer already exists 
8fd544ed9d84: Layer already exists 
dce74d3b1b1f: Layer already exists 
cb24fdf7aeef: Pushing [==================================================>] 20.11 MB
ea5d2895543b: Layer already exists 
9fc968a57e8a: Layer already exists 
55aa97f20891: Layer already exists 
65f3b0435c42: Pushing [==================================================>]   131 MB
unauthorized: authentication required

The message appears after a long while. I've seen this in the logs:

2016-09-02_08:22:27.87029 time="2016-09-02T08:22:27.870257413Z" level=error msg="token not to be used before 1472804178 or after 1472804483 - currently 1472804547" 
2016-09-02_08:22:27.87053 time="2016-09-02T08:22:27.870491895Z" level=warning msg="error authorizing context: invalid token" environment=production go.version=go1.5.4 http.request.host="<server>" http.request.id=d545298d-bcd4-4574-a556-423260cae492 http.request.method=PATCH http.request.remoteaddr=<ipaddr> http.request.uri="/v2/<group>/<project>/blobs/uploads/8d0365ee-a69a-408a-b3c4-615e81994c04?_state=HD3wyVfDw3hW1De-DbMZR0p1SBJHOoSaHSDluvDUTpZ7Ik5hbWUiOiJtdWx0aXRlbmFudC9wb3N0Z3JlcyIsIlVVSUQiOiI4ZDAzNjVlZS1hNjlhLTQwOGEtYjNjNC02MTVlODE5OTRjMDQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTYtMDktMDJUMDg6MTY6MjUuMjcyNTAzMDE5WiJ9" http.request.useragent="docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/4.4.0-36-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))" instance.id=c6b0612f-4fd7-4c56-93a4-7549e76c30d8 service=registry vars.name="<group>/<project>" vars.uuid=8d0365ee-a69a-408a-b3c4-615e81994c04 version=v2.4.1 

I'm quite sure I'm seeing a duplicate of issue #17890 (closed), that was closed with commit #1d94757c that made the Container Registry token expire delay configurable. Posted for people arriving here at random so that they find faster how to proceeed :)

Issue still exists.

darshan@test:~$ sudo docker login registry.gitlab.com [sudo] password for darshan: Username (sdfsdf): Password: Login Succeeded darshan@Endoluxe:~$ sudo docker push registry.gitlab.com/xxx/xxx:build The push refers to a repository [registry.gitlab.com/xxx/xxx] 10be4f443b4c: Pushing 2.934 GB 2a0f842b5660: Layer already exists acedbb91a412: Layer already exists 67383556b4f3: Pushing 1.727 GB 67852b7d2cf5: Layer already exists f1f25d8c6e56: Layer already exists 33a212e8aa56: Layer already exists eaaf72983326: Layer already exists a85f35566a26: Layer already exists unauthorized: authentication required darshan@test:~$

Same problem here:

[capitanio@cdprh73 docker]$ docker login xxx:5001 Username (mcapitanio): Password: Login Succeeded [capitanio@cdprh73 docker]$ docker-compose push Pushing galileo-anagrafica-rapporti-services (gitlab.cdp.it:5001/xxx/xxx:local)... The push refers to a repository [gitlab.cdp.it:5001/xxx/xxx] dceb6a7cb17f: Preparing b6f1cc9061bf: Preparing 9d82ae54a5cf: Preparing 4013abe830ae: Preparing b533bd07213e: Preparing 1489b9753207: Waiting fdf3e3585a6e: Waiting 445bb117c8fc: Waiting 0c32b427b0db: Waiting f1fccf59de9c: Waiting aafb3efdaa1e: Waiting d4def48e1f46: Waiting e1f0e0d05245: Waiting 7fb72a5c13b6: Waiting b8e531613021: Waiting 2ad6ba0f254d: Waiting 690e2e32726f: Waiting 24f1f163b88e: Waiting 45d7ba89d79d: Waiting bd5906b02180: Waiting 8675e0f87a5f: Waiting f61ca3c6ccbc: Waiting e15700a90287: Waiting cf21d83208b6: Waiting 50e3a7eeb2fc: Waiting 40fe9f2649f9: Waiting fbc0f3f20f9f: Waiting 0aeb287b1ba9: Waiting ERROR: unauthorized: authentication required

Please, we have also an Enterprise Edition, this functionality has to be reliable.

Same problem here.

Same problem here!

Same issue here. Using Docker 1.12.3 and gitlab-ce omnibus 8.14.0

Please read my previous comment carefully.

I'm quite sure I'm seeing a duplicate of issue #17890 (closed), that was closed with commit #1d94757c that made the Container Registry token expire delay configurable. Posted for people arriving here at random so that they find faster how to proceeed :)

@sgala I did read the comment before posting. Thanks for it, by the way.

And i had tried it with 600 min as timeout, but still have the issue, even with a small container being pushed a second after the login command. Forgot to say that in my last comment.

I strongly believe there is more to this issue than that.

Can I change these settings also for gitlab.com? We are using free runners.

Same problem with "unauthorized: authentication required" here ... So I must setup own registry

@scippio: Yeah, I gave up on gitlab registry: gitlab_rails['registry_enabled'] = false

Then I picked up one of the many registry options from docker hub and in just a few minutes I had it working perfectly.

A week battling gitlab's registry vs. 30 min to set up an external one; easy peasy.

Since this issue was opened, I stopped update GitLab on the 22nd of every month. I always wait the .2 or .3 of each version. This was the third time I had the whole company workflow interrupted by errors in the 22nd release. And apparently, I will not be able to apply the next Monday's security update . Too bad

Seeing this problem today after many timeouts. Looking into setting up another registry somewhere else for the time being (possibly permanently).

I used image docker:latest instead of image docker:git. It's working now.

What is the workaround? I'm using shell executor and thus solution of previous comment doest not apply.

For me the solution was to enable the Container Registry in the project settings.

Still no solution for this? I'm experiencing the same problem on gitlab ce 8.16.0 ca1076ee

upgraded my gitlab-ce version to 65cdbdd8, same problem persist

This issue still persists for me, on the latest versions of GitLab (9.4.0) and Docker (17.06.0-ce-mac19).

I was experiencing same problem and the cause was gitlab behind proxy and wrong setting of repository address in gitlab config (http instead of https). Solved using https://gitlab.com/gitlab-com/infrastructure/issues/256

@joe19 I am also using a reverse proxy which is likely causing this issue. I didn't find the solution in the issue you linked to. Can you quote the solution and comment here?

reopened

I set registry_nginx['proxy_set_headers'] = { "Host" => "registry.mydomain.com" } and then my another problem was that I had "http" registry address in config, not "https".

I used wireshark to debug HTTP coomunication of docker daemon and found that it connects to wrong host or wrong port.

Same problem here. Any clear fix known ? it has been opened 13 month ago now.

• It fails after 7 minutes and a few seconds, so not a '5min timeout' problem
• I haven't set any proxy : so no proxy config problem

a217c7618e5c: Layer already exists 
2c630cb2fc32: Layer already exists 
e54ab0c61e68: Layer already exists 
20a7a583cb41: Layer already exists 
795bb37617dd: Layer already exists 
aeed236eb430: Layer already exists 
d86c7816f57b: Layer already exists 
ee8ec558a623: Layer already exists 
fd9692221e54: Layer already exists 
21f2f9be1327: Layer already exists 
bce96745b9ca: Layer already exists 
e4f596b42f61: Layer already exists 
def5c2f989f7: Layer already exists 
702260f1094c: Layer already exists 
d2b01b4f3e42: Layer already exists 
0d5b4cefb4fd: Pushing [==================================================>] 42.77 MB
c8c6d352c96e: Layer already exists 
5a151fa4136e: Layer already exists 
8747705a74d9: Pushing [==================================================>] 33.01 MB
adfd7c98cdf6: Layer already exists 
354b4ef9664b: Pushing [==================================================>] 13.01 MB
2da371244fc1: Layer already exists 
315db89f9acb: Layer already exists 
3d2582fdede2: Layer already exists 
53da01089a82: Layer already exists 
0fd7a67d49b3: Layer already exists 
f0c367fa8636: Layer already exists 
ddd6dcab19ff: Pushing [==========>                                        ] 40.68 MB/200.2 MB
2c40c66f7667: Pushing [=================>                                 ] 42.39 MB/123.5 MB
unauthorized: authentication required

=> EDIT :

In fact, it was due to the Authorization timeout. Even if the error happens 2 minutes after the effective timeout. (Analyzing '/var/log/gitlab/registry/' helped me understand that)

After editing the Authorization timeout as suggested by @grzesiek , everything went fine.

Thanks

@nicholastulach Authorization timeout is now configurable through the UI -> Settings -> Pipelines page.

closed

added CI/CD label

reopened

Same problem here. I can push some images, but some images always fail, irrelevant to login before.

I have managed to solve it (most of the time) by setting Authorization token duration in admin >> application_settings >> Container Registry

moved to gitlab#5540

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue