CI Runners Fails to Clone Repo after 8.12.0 upgrade
Summary
CI builds are failing to clone the repo with a 500 error. These builds were working prior to the upgrade.
Running with gitlab-ci-multi-runner 1.6.0 (01b3ea1)
Using Docker executor with image node:4-wheezy ...
Pulling docker image rethinkdb:latest ...
Starting service rethinkdb:latest ...
Waiting for services to be up and running...
Pulling docker image node:4-wheezy ...
Running on runner-ffe79177-project-499-concurrent-0 via host.domain.com...
Cloning repository...
Cloning into '/builds/group/project'...
remote: Internal server error
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@host.domain.com/group/project.git/': The requested URL returned error: 500
ERROR: Build failed: exit code 1Also users that have previously cloned repos using the https protocol are unable to execute pull requests, ssh protocol is working fine.
$ git pull
fatal: unable to access 'https://host.domain.com/group/project.git': The requested URL returned error: 500(Summarize the bug encountered concisely)
Steps to reproduce
- Setup a gitlab-ci-multi-runner and register it with GitLab as a global runner
- Setup a project to perform a build
- Push a branch up to GitLab to trigger the build
(How one can reproduce the issue - this is very important)
Expected behavior
- CI Runner should be able to clone the repo that it was triggered to do the build for
- Users that were previously able to clone the repo via HTTPS should be able to successfully perform a pull (What you should see instead)
Actual behavior
500 error is encountered (What actually happens)
Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)
From the Admin page on GitLabEE
GitLab 8.12.0-ee
GitLab Shell 3.6.0
GitLab Workhorse 0.8.2
GitLab API v3
Git 2.7.4
Ruby 2.3.1p112
Rails 4.2.7.1
PostgreSQL 9.2.18$ /opt/gitlab/embedded/bin/gitlab-workhorse --version
gitlab-workhorse v0.8.2-20160922.154937Output of checks
Results of GitLab application Check
$ sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...
GitLab Shell version >= 3.6.0 ? ... OK (3.6.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
4/4 ... ok
4/5 ... ok
4/7 ... ok
4/8 ... ok
4/9 ... ok
4/10 ... ok
4/13 ... ok
4/14 ... ok
51/16 ... ok
4/18 ... ok
39/19 ... ok
4/20 ... ok
42/21 ... ok
42/22 ... ok
8/24 ... ok
52/25 ... ok
52/26 ... ok
4/27 ... ok
52/30 ... ok
52/31 ... ok
4/32 ... ok
4/33 ... ok
4/34 ... ok
4/36 ... ok
8/38 ... ok
8/46 ... ok
2/54 ... ok
46/56 ... ok
46/57 ... ok
45/58 ... ok
21/60 ... ok
21/61 ... ok
9/62 ... ok
20/63 ... ok
35/65 ... ok
9/68 ... ok
33/72 ... ok
21/73 ... ok
21/74 ... ok
33/75 ... ok
36/76 ... ok
46/77 ... ok
4/78 ... ok
39/81 ... ok
4/82 ... ok
39/83 ... ok
39/84 ... ok
4/85 ... ok
39/86 ... ok
39/87 ... ok
4/88 ... ok
39/89 ... ok
39/90 ... ok
39/91 ... ok
39/92 ... ok
39/93 ... ok
39/95 ... ok
39/96 ... ok
39/97 ... ok
39/98 ... ok
39/99 ... ok
40/100 ... ok
33/101 ... ok
49/102 ... ok
49/104 ... ok
49/105 ... ok
47/106 ... ok
9/107 ... ok
46/108 ... ok
21/111 ... ok
20/113 ... ok
20/114 ... ok
49/119 ... ok
61/120 ... ok
61/121 ... ok
61/122 ... ok
61/123 ... ok
61/124 ... ok
61/125 ... ok
41/126 ... ok
41/127 ... ok
41/128 ... ok
41/129 ... ok
41/130 ... ok
41/131 ... ok
41/132 ... ok
41/133 ... ok
41/134 ... ok
61/135 ... ok
30/136 ... ok
96/139 ... ok
78/140 ... ok
78/141 ... ok
3/142 ... ok
107/143 ... ok
33/145 ... ok
33/146 ... ok
33/147 ... ok
33/148 ... ok
8/150 ... ok
57/153 ... ok
57/154 ... ok
106/158 ... ok
106/159 ... ok
106/160 ... ok
106/161 ... ok
106/162 ... ok
107/164 ... ok
108/165 ... ok
108/166 ... ok
108/167 ... ok
110/168 ... ok
107/170 ... ok
108/171 ... ok
108/172 ... ok
105/173 ... ok
106/176 ... ok
33/177 ... ok
8/178 ... ok
94/179 ... ok
111/180 ... repository is empty
61/181 ... ok
51/182 ... ok
51/183 ... ok
144/184 ... ok
105/185 ... ok
33/186 ... ok
31/187 ... ok
96/189 ... ok
105/190 ... ok
109/195 ... ok
47/196 ... ok
9/197 ... ok
51/200 ... ok
8/204 ... ok
8/205 ... ok
118/207 ... ok
34/208 ... ok
8/211 ... ok
105/212 ... ok
2/214 ... ok
2/216 ... ok
2/217 ... ok
118/218 ... ok
30/219 ... ok
2/221 ... ok
78/228 ... repository is empty
124/229 ... ok
124/230 ... ok
124/233 ... ok
124/234 ... ok
124/235 ... ok
8/237 ... ok
61/238 ... ok
61/239 ... ok
181/240 ... ok
61/241 ... ok
61/242 ... ok
61/243 ... ok
51/245 ... ok
61/246 ... ok
127/249 ... ok
9/250 ... ok
130/251 ... ok
61/252 ... ok
108/253 ... ok
131/261 ... ok
136/263 ... ok
136/264 ... ok
136/265 ... ok
136/266 ... repository is empty
136/267 ... repository is empty
136/268 ... repository is empty
136/269 ... repository is empty
136/270 ... repository is empty
136/271 ... repository is empty
136/272 ... repository is empty
136/273 ... repository is empty
51/275 ... ok
108/277 ... ok
108/278 ... repository is empty
108/279 ... ok
51/280 ... ok
106/281 ... ok
106/282 ... ok
106/283 ... ok
61/284 ... ok
9/285 ... ok
33/288 ... ok
2/289 ... ok
106/290 ... ok
106/291 ... ok
106/292 ... ok
106/293 ... ok
33/294 ... ok
43/295 ... ok
43/296 ... ok
144/298 ... ok
144/299 ... ok
96/301 ... ok
96/303 ... ok
61/304 ... ok
105/305 ... ok
106/306 ... ok
2/307 ... ok
61/308 ... ok
61/309 ... ok
61/310 ... ok
61/311 ... ok
181/312 ... ok
61/314 ... ok
96/317 ... ok
61/318 ... ok
51/319 ... ok
51/320 ... ok
51/321 ... ok
8/322 ... ok
150/326 ... ok
114/327 ... ok
61/328 ... ok
96/329 ... ok
96/330 ... ok
130/332 ... ok
30/333 ... ok
9/335 ... ok
131/336 ... ok
150/337 ... ok
150/338 ... ok
61/340 ... ok
153/341 ... ok
164/342 ... ok
164/343 ... ok
164/344 ... ok
165/345 ... ok
165/346 ... ok
165/348 ... repository is empty
150/349 ... ok
61/351 ... ok
61/352 ... ok
61/353 ... ok
150/354 ... ok
61/356 ... ok
33/357 ... ok
33/358 ... ok
61/359 ... ok
33/360 ... ok
51/361 ... ok
51/362 ... ok
51/363 ... ok
51/364 ... ok
51/365 ... ok
51/366 ... ok
51/367 ... ok
51/368 ... ok
51/369 ... ok
61/370 ... ok
131/371 ... ok
51/372 ... ok
51/373 ... ok
51/374 ... ok
51/375 ... ok
150/376 ... ok
153/377 ... ok
33/378 ... ok
33/379 ... ok
110/380 ... ok
43/381 ... ok
43/382 ... ok
124/383 ... ok
96/384 ... ok
174/385 ... ok
150/386 ... ok
150/390 ... ok
176/392 ... ok
176/394 ... ok
33/397 ... ok
61/400 ... ok
61/401 ... ok
155/402 ... ok
61/403 ... ok
150/404 ... ok
8/406 ... ok
61/407 ... ok
61/409 ... ok
61/410 ... ok
153/411 ... ok
180/412 ... ok
61/413 ... ok
180/418 ... repository is empty
180/420 ... ok
8/421 ... ok
61/423 ... ok
150/424 ... ok
61/425 ... ok
61/426 ... ok
61/427 ... ok
61/428 ... ok
61/429 ... ok
96/430 ... ok
61/431 ... ok
61/433 ... ok
61/434 ... ok
150/435 ... ok
61/436 ... ok
33/437 ... ok
33/438 ... ok
33/439 ... ok
150/440 ... ok
181/441 ... ok
181/442 ... ok
181/443 ... ok
61/444 ... ok
181/445 ... ok
181/446 ... ok
33/448 ... ok
61/449 ... ok
150/452 ... ok
153/455 ... ok
153/456 ... ok
153/457 ... ok
150/458 ... ok
153/459 ... ok
96/460 ... ok
110/461 ... ok
61/463 ... ok
61/464 ... ok
61/465 ... ok
150/468 ... ok
33/469 ... repository is empty
43/472 ... ok
130/473 ... ok
130/474 ... ok
43/483 ... ok
131/491 ... ok
185/493 ... ok
61/494 ... ok
158/497 ... ok
131/498 ... ok
49/499 ... ok
49/500 ... ok
63/501 ... ok
202/503 ... ok
29/504 ... ok
192/505 ... ok
29/506 ... ok
49/507 ... ok
105/508 ... ok
63/510 ... ok
96/511 ... ok
74/512 ... ok
96/513 ... ok
96/514 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 2
Try fixing it:
sudo service gitlab stop
sudo pkill -u git -f sidekiq
sleep 10 && sudo pkill -9 -u git -f sidekiq
sudo service gitlab start
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP users with access to your GitLab server (only showing the first 100 results)
<< Omitted LDAP User List >>
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
4/4 ... yes
4/5 ... yes
4/7 ... yes
4/8 ... yes
4/9 ... yes
4/10 ... yes
4/13 ... yes
4/14 ... yes
51/16 ... yes
4/18 ... yes
39/19 ... yes
4/20 ... yes
42/21 ... yes
42/22 ... yes
8/24 ... yes
52/25 ... yes
52/26 ... yes
4/27 ... yes
52/30 ... yes
52/31 ... yes
4/32 ... yes
4/33 ... yes
4/34 ... yes
4/36 ... yes
8/38 ... yes
8/46 ... yes
2/54 ... yes
46/56 ... yes
46/57 ... yes
45/58 ... yes
21/60 ... yes
21/61 ... yes
9/62 ... yes
20/63 ... yes
35/65 ... yes
9/68 ... yes
33/72 ... yes
21/73 ... yes
21/74 ... yes
33/75 ... yes
36/76 ... yes
46/77 ... yes
4/78 ... yes
39/81 ... yes
4/82 ... yes
39/83 ... yes
39/84 ... yes
4/85 ... yes
39/86 ... yes
39/87 ... yes
4/88 ... yes
39/89 ... yes
39/90 ... yes
39/91 ... yes
39/92 ... yes
39/93 ... yes
39/95 ... yes
39/96 ... yes
39/97 ... yes
39/98 ... yes
39/99 ... yes
40/100 ... yes
33/101 ... yes
49/102 ... yes
49/104 ... yes
49/105 ... yes
47/106 ... yes
9/107 ... yes
46/108 ... yes
21/111 ... yes
20/113 ... yes
20/114 ... yes
49/119 ... yes
61/120 ... yes
61/121 ... yes
61/122 ... yes
61/123 ... yes
61/124 ... yes
61/125 ... yes
41/126 ... yes
41/127 ... yes
41/128 ... yes
41/129 ... yes
41/130 ... yes
41/131 ... yes
41/132 ... yes
41/133 ... yes
41/134 ... yes
61/135 ... yes
30/136 ... yes
96/139 ... yes
78/140 ... yes
78/141 ... yes
3/142 ... yes
107/143 ... yes
33/145 ... yes
33/146 ... yes
33/147 ... yes
33/148 ... yes
8/150 ... yes
57/153 ... yes
57/154 ... yes
106/158 ... yes
106/159 ... yes
106/160 ... yes
106/161 ... yes
106/162 ... yes
107/164 ... yes
108/165 ... yes
108/166 ... yes
108/167 ... yes
110/168 ... yes
107/170 ... yes
108/171 ... yes
108/172 ... yes
105/173 ... yes
106/176 ... yes
33/177 ... yes
8/178 ... yes
94/179 ... yes
111/180 ... yes
61/181 ... yes
51/182 ... yes
51/183 ... yes
144/184 ... yes
105/185 ... yes
33/186 ... yes
31/187 ... yes
96/189 ... yes
105/190 ... yes
109/195 ... yes
47/196 ... yes
9/197 ... yes
51/200 ... yes
8/204 ... yes
8/205 ... yes
118/207 ... yes
34/208 ... yes
8/211 ... yes
105/212 ... yes
2/214 ... yes
2/216 ... yes
2/217 ... yes
118/218 ... yes
30/219 ... yes
2/221 ... yes
78/228 ... yes
124/229 ... yes
124/230 ... yes
124/233 ... yes
124/234 ... yes
124/235 ... yes
8/237 ... yes
61/238 ... yes
61/239 ... yes
181/240 ... yes
61/241 ... yes
61/242 ... yes
61/243 ... yes
51/245 ... yes
61/246 ... yes
127/249 ... yes
9/250 ... yes
130/251 ... yes
61/252 ... yes
108/253 ... yes
131/261 ... yes
136/263 ... yes
136/264 ... yes
136/265 ... yes
136/266 ... yes
136/267 ... yes
136/268 ... yes
136/269 ... yes
136/270 ... yes
136/271 ... yes
136/272 ... yes
136/273 ... yes
51/275 ... yes
108/277 ... yes
108/278 ... yes
108/279 ... yes
51/280 ... yes
106/281 ... yes
106/282 ... yes
106/283 ... yes
61/284 ... yes
9/285 ... yes
33/288 ... yes
2/289 ... yes
106/290 ... yes
106/291 ... yes
106/292 ... yes
106/293 ... yes
33/294 ... yes
43/295 ... yes
43/296 ... yes
144/298 ... yes
144/299 ... yes
96/301 ... yes
96/303 ... yes
61/304 ... yes
105/305 ... yes
106/306 ... yes
2/307 ... yes
61/308 ... yes
61/309 ... yes
61/310 ... yes
61/311 ... yes
181/312 ... yes
61/314 ... yes
96/317 ... yes
61/318 ... yes
51/319 ... yes
51/320 ... yes
51/321 ... yes
8/322 ... yes
150/326 ... yes
114/327 ... yes
61/328 ... yes
96/329 ... yes
96/330 ... yes
130/332 ... yes
30/333 ... yes
9/335 ... yes
131/336 ... yes
150/337 ... yes
150/338 ... yes
61/340 ... yes
153/341 ... yes
164/342 ... yes
164/343 ... yes
164/344 ... yes
165/345 ... yes
165/346 ... yes
165/348 ... yes
150/349 ... yes
61/351 ... yes
61/352 ... yes
61/353 ... yes
150/354 ... yes
61/356 ... yes
33/357 ... yes
33/358 ... yes
61/359 ... yes
33/360 ... yes
51/361 ... yes
51/362 ... yes
51/363 ... yes
51/364 ... yes
51/365 ... yes
51/366 ... yes
51/367 ... yes
51/368 ... yes
51/369 ... yes
61/370 ... yes
131/371 ... yes
51/372 ... yes
51/373 ... yes
51/374 ... yes
51/375 ... yes
150/376 ... yes
153/377 ... yes
33/378 ... yes
33/379 ... yes
110/380 ... yes
43/381 ... yes
43/382 ... yes
124/383 ... yes
96/384 ... yes
174/385 ... yes
150/386 ... yes
150/390 ... yes
176/392 ... yes
176/394 ... yes
33/397 ... yes
61/400 ... yes
61/401 ... yes
155/402 ... yes
61/403 ... yes
150/404 ... yes
8/406 ... yes
61/407 ... yes
61/409 ... yes
61/410 ... yes
153/411 ... yes
180/412 ... yes
61/413 ... yes
180/418 ... yes
180/420 ... yes
8/421 ... yes
61/423 ... yes
150/424 ... yes
61/425 ... yes
61/426 ... yes
61/427 ... yes
61/428 ... yes
61/429 ... yes
96/430 ... yes
61/431 ... yes
61/433 ... yes
61/434 ... yes
150/435 ... yes
61/436 ... yes
33/437 ... yes
33/438 ... yes
33/439 ... yes
150/440 ... yes
181/441 ... yes
181/442 ... yes
181/443 ... yes
61/444 ... yes
181/445 ... yes
181/446 ... yes
33/448 ... yes
61/449 ... yes
150/452 ... yes
153/455 ... yes
153/456 ... yes
153/457 ... yes
150/458 ... yes
153/459 ... yes
96/460 ... yes
110/461 ... yes
61/463 ... yes
61/464 ... yes
61/465 ... yes
150/468 ... yes
33/469 ... yes
43/472 ... yes
130/473 ... yes
130/474 ... yes
43/483 ... yes
131/491 ... yes
185/493 ... yes
61/494 ... yes
158/497 ... yes
131/498 ... yes
49/499 ... yes
49/500 ... yes
63/501 ... yes
202/503 ... yes
29/504 ... yes
192/505 ... yes
29/506 ... yes
49/507 ... yes
105/508 ... yes
63/510 ... yes
96/511 ... yes
74/512 ... yes
96/513 ... yes
96/514 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 111
Checking GitLab ... Finished
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)
Results of GitLab environment info
System information
System: Ubuntu 14.04
Current User: git
Using RVM: no
Ruby Version: 2.3.1p112
Gem Version: 2.6.6
Bundler Version:1.13.1
Rake Version: 10.5.0
Sidekiq Version:4.1.4
GitLab information
Version: 8.12.0-ee
Revision: 0af2753
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 9.2.18
URL: https://git.ibaset.com
HTTP Clone URL: https://git.ibaset.com/some-group/some-project.git
SSH Clone URL: git@git.ibaset.com:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: yes
Using Omniauth: no
GitLab Shell
Version: 3.6.0
Repository storage paths:
- default: /data00/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production)
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
Activity
/cc @ayufan
Also, a discussion on this issue was started on the CE side yesterday. Same problem over there.
Trying to clone a repo via HTTPS is broken the same as CI. SSH works a treat
$ git clone https://git.ibaset.com/group/project.git Cloning into 'projcet'... remote: Internal server error fatal: unable to access 'https://git.ibaset.com/group/project.git/': The requested URL returned error: 500 $ git clone git@git.ibaset.com:group/project.git Cloning into 'project'... remote: Counting objects: 241, done. remote: Compressing objects: 100% (135/135), done. remote: Total 241 (delta 132), reused 178 (delta 96) Receiving objects: 100% (241/241), 51.73 KiB | 0 bytes/s, done. Resolving deltas: 100% (132/132), done.Tailing the log in gilab here is what I am seeing
==> /var/log/gitlab/gitlab-workhorse/current <== 2016-09-23_16:54:52.96779 2016/09/23 09:54:52 error: preAuthorizeHandler: newUpstreamRequest: newRequest: read Secret.Path: open ./.gitlab_workhorse_secret: no such file or directory 2016-09-23_16:54:52.96783 git.domain.com @ - - [2016-09-23 09:54:52.967429301 -0700 PDT] "GET /group/project.git/info/refs?service=git-upload-pack HTTP/1.1" 500 22 "" "git/2.8.3" 0.000342it does appear that a file with that name in some form does exist, just don't know where the workhorse is actually looking and the filename does not start with a
.$ cat /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret <secret>Edited by Drew Blessing
@kstover Are you setting
proxy_headers_timeoutin/etc/gitlab/gitlab.rbby chance? See:Correct. I added the following to /etc/gitlab/gitlab.rb
Workhorse
gitlab_workhorse['proxy_headers_timeout'] = '10m'
I had to introduce the timeout setting because we were getting a 502 when trying to view merge requests with an extremely large amount of changes. BTW: this was a very hard setting to find, I came across it here https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/610/diffs. This might be a good setting to document as I am sure other people may come across the same.
Error I was receiving that led me to add the timeout setting ==> /var/log/gitlab/gitlab-workhorse/current <== 2016-05-27_22:06:08.66099 2016/05/27 15:06:08 error: proxyRoundTripper: GET "/group/project/merge_requests/1319" failed with: "net/http: timeout awaiting response headers"
Please register or sign in to reply