Random issues with `HTTP Basic: Access denied` on repo cloning
Summary
Running with gitlab-ci-multi-runner 1.10.4 (b32125f)
Using Docker executor with image docker:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Waiting for services to be up and running...
Pulling docker image docker:latest ...
Running on runner-4e4528ca-project-1693965-concurrent-0 via runner-4e4528ca-machine-1490105937-01f38f71-digital-ocean-4gb...
Cloning repository...
Cloning into '/builds/GROUP/PROJECT'...
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.com/GROUP/PROJECT.git/'
ERROR: Build failed: exit code 1May be connected to #2244
Steps to reproduce
How one can reproduce the issue - this is very important
No clue, random occurances, I can provide more details in private mode
What actually happens
This happens randomly, simple job restart helps
What you should see instead
Successful git clone
Environment description
Applicable for shared runner as well as for specific runners
Used GitLab Runner version
shared runners, usually 1.10.4
specific runners on 1.11.1
Activity
I've seen another one like this, just a minute ago. @ayufan are our runners having issues?
-
Also #2263 (closed)
I'm not sure. We should not have. @briann Are you aware of any problems with Rack Attack, I know that we had some changes there.
@ayufan I'm not aware of any Rack Attack issues at the moment. The problems we saw last week were related to the new patch that limits authentication from multiple IPs. I've verified that is still disabled.
mentioned in issue gitlab-com/infrastructure#1441 (closed)
It is probably caused by: https://gitlab.com/gitlab-com/infrastructure/issues/1441#note_26005965
mentioned in issue #2244
Right now i have the same issue booth running shared and specific runner
Running with gitlab-ci-multi-runner 9.0.1 (a3da309) on docker-auto-scale (e11ae361) Using Docker executor with image docker:1.13 ... Starting service docker:1.13-dind ... Pulling docker image docker:1.13-dind ... Using docker image docker:1.13-dind ID=sha256:0534a070b830fbbcd8e870b53e4e7de0aa3ae4b7627ae1913604a9881a1f5fcc for service docker... Waiting for services to be up and running... Using docker image sha256:176ce157c2cd571c4525a3b946f371ebac6b150d28d58baacbddf4cbbf64b5b0 ID=sha256:176ce157c2cd571c4525a3b946f371ebac6b150d28d58baacbddf4cbbf64b5b0 for predefined container... Pulling docker image docker:1.13 ... Using docker image docker:1.13 ID=sha256:9ec0bbbc650d5a7614227c80231b99ea90bce075f327fcbf1fd40e33aef15421 for build container... Running on runner-e11ae361-project-3048430-concurrent-0 via runner-e11ae361-machine-1491718220-10831d18-digital-ocean-2gb... Cloning repository... Cloning into '/builds/cimenx/docker-ros-deep-perception'... remote: HTTP Basic: Access denied fatal: Authentication failed for 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.com/cimenx/docker-ros-deep-perception.git/' ERROR: Job failed: exit code 1I have the same issues!
Running with gitlab-ci-multi-runner 9.2.0 (adfc387) on my-runner-two (aa727638) Using Docker executor with image my-php-sql:5.6 ... Starting service mysql:5.6 ... Using locally found image version due to if-not-present pull policy Using docker image mysql:5.6 ID=sha256:ed7b6c642b9dde55c6c86bc58b5d3b571e1a9327c49e32a49433cfe5e792b24d for mysql service... Waiting for services to be up and running... Using docker image sha256:ddf58f47bb0c714bc7f4f908c794e15cef6cf8c69532d1014af94c54e693496e for predefined container... Using locally found image version due to if-not-present pull policy Using docker image my-php-sql:5.6 ID=sha256:bc8dc818b8a7abb30a9826f514351cffbf259cd2944cb333df693632dfb92259 for build container... Running on runner-aa727638-project-173-concurrent-0 via gitlab-runner... Cloning repository... Cloning into '/builds/my/web'... remote: HTTP Basic: Access denied fatal: Authentication failed for 'https://gitlab.my.de/my/web.git/' ERROR: Job failed: exit code 1I already readded the runner. The error just appears since 3 weeks permantly, also on job retries. I´m using GitLab Community Edition 9.2.1
Edited by username-removed-1350878@develth can you grep your
production.logfile for "Rack_Attack" and see if there is a Rack Attack ban on the runner IP? Rack Attack bans would last only an hour by default.@briann can´t find a log entry with this keyword. The Failure is always thrown - even after a few days of "no running".
Here a syslog of the issue. Job 35258503 runs for a couple minutes successfully. Then 3 runs in a row fail because of this issue (job 35258504, 35258505, and 35258506). Job 35258507 then works again.
Maybe somebody could have a look into the gitlab.com logs to get more insight into what is going wrong here.
Edited by username-removed-723668
