Gitlab runner randomly can't clone repository : HTTP Basic: Access denied

Summary

While doing a pipeline, some task randomly fails due to gitlab-ci-token which seems to be unable to clone the repository.

Relevant logs and/or screenshots

Running with gitlab-ci-multi-runner 1.11.1 (a67a225)
  on ci-runner01 (2591046a)
Using Shell executor...
Running on ci-runner01..
Fetching changes...
Removing vendor/
HEAD is now at dd42fa1 Testing gitlab CI
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.com/PROJECT/www.repo.com.git/'
Fetching changes...
HEAD is now at dd42fa1 Testing gitlab CI
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.com/PROJECT/www.repo.com.git/'
Fetching changes...
HEAD is now at dd42fa1 Testing gitlab CI

Environment description

We are using a custom runner on gitlab.com Shell executor

Used GitLab Runner version

Git revision: a67a225
Git branch:   1-11-stable
GO version:   go1.7.5
Built:        Fri, 03 Mar 2017 12:35:58 +0000
OS/Arch:      linux/amd64```

We checked that the project CI token is the same that the one we used to register the runner.

We checked that the project CI token is the same that the one we used to register the runner.

To make this clear, because it can be a source of problems in the future, in CI we're using three types of tokens:

• registration token - there is one global for whole GitLab installation (used for registering shared runners) and one individual per project (used for registering specific runners for projects). This one is used only for registration (at the beginning it was the only token but this was changed a long time ago).
• runner's token - this is generated for each registered Runner and stored in Runner's config.toml file. This one is used for authenticating Runner when it is - for example - asking for a job to execute. For a long time it was also used as the value of $CI_BUILD_TOKEN to authenticate git clone/fetch.
• job's token - this is generated for each started job. It was introduced some time ago and at the beginning used for git clone/fetch authentication. After changes in CI permission model in 8.12 it was also possible to use for accessing images from other private/internal projects. Since some time it is used to authenticate artifacts uploads/downloads by Runner. And in 9.0 it will be the only token which authenticates job's update.

Now, back to your problem. Are you using gitlab.com or is this left only as an example hostname in the log above? If you are using your own GitLab installation can you write which version are you using?

added support label

Hi @tmaczukin and thanks for your message.

We are using gitlab.com for hosting our code but we call our own ci-runners. We figured out that, when we cancel a pipeline, it leaves the build folder on the server intact. That's the main reason why the following pipeline failed because we made test and changed our runner token many times.

However, it has been cleaned up and we still have issue regarding the fetch/clone action and random jobs of our pipelines.

Dunno if you can, as an admin, access to the pipeline ID 7032024 but we can give you access to our project if needed.

There is something i don't understand in your answer.

I can see the job token being used in our project since it changes the origin URL of every clone (on every job build). But why does it fails randomly ? The failure is not always on the same job..

A hypothesis we are investigating :

• We are using the shell executor
• We understood the gitlab runner works this way for a single folder : EXECUTING TASK 1 > WIPE FOLDER > EXECUTING TASK 2 > WIPE FOLDER > ...
• We think, sometimes, it doesn't wipe succesfully the folder and the next task began by a git fetch on an expired token.

I have the same issue lately, it randomly cannot clone project, just have to retry the build and it works.

I've just tried with a docker executor, same issue.

Same on Shared runners also. Seems to be a global issue.

changed title from Gitlab runner randomly can't clone repository to Gitlab runner randomly can't clone repository : HTTP Basic: Access denied

@jean-francois.labbe Does your problem exists also for GitLab.com? Or do you use your own GitLab installation?

On GitLab.com such random bugs recently are generated probably by gitlab-com/infrastructure#1441 (closed).

@tmaczukin we use gitlab.com

https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2248 https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2263

it seems to be better now

Confirming that things are getting better since monday but got almost 7 days of huge build problems.

closed

Definitely related to gitlab-com/infrastructure#1441 (closed). I think we can close this issue at this moment :)

Can we reopen this? I am getting these issues all of a sudden. I've tried restarting my gitlab-runner, but the problem persists!

I have the same Problem aswell (first occurred, 3 days ago). Sometimes restarting the job fixes the issue, sometimes a new push or restarting the runner helps...

Same here. Having the same problem on Gitlab.com's shared runners.

Running with gitlab-ci-multi-runner 9.5.0 (413da38)
  on docker-auto-scale (e11ae361)
Using Docker executor with image ruby:2.3 ...
Using docker image sha256:bb128c3ec41d455be22a35510dc9c38287c1889f62de385041c8e1d502fb6f12 for predefined container...
Pulling docker image ruby:2.3 ...
Using docker image ruby:2.3 ID=sha256:30e38ebeba79ac2d62bcff6b094e6478098cc1da0a1df14d67211477a31fa3f9 for build container...
Running on runner-e11ae361-project-3507174-concurrent-0 via runner-e11ae361-machine-1503881124-390d1219-digital-ocean-2gb...
Cloning repository...
Cloning into '/builds/docbot.md/hippocrates'...
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.com/docbot.md/hippocrates.git/'
ERROR: Job failed: exit code 1