401 Unauthorized access when cloning or pushing after upgrade from 8.11 to 8.12, which surface to the user as 500 Internal Server Error
Hello,
Today after an upgrade to 8.12 from 8.11 (not sure which one, I think version 3), I noticed an issue with cloning and pushing to the git repositories. I've been fighting with this all day long and can't find a solution. The only thing that pops ups in the logs is this, in production.log:
Started POST "/ci/api/v1/builds/register.json" for 192.168.122.20 at 2016-09-23 15:33:25 +0200
Started GET "/namespace/project.git/info/refs?service=git-upload-pack" for 192.168.122.20 at 2016-09-23 15:33:25 +0200
Processing by Projects::GitHttpController#info_refs as */*
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"namespace", "project_id"=>"project.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 127ms (Views: 0.8ms | ActiveRecord: 13.4ms)
Started GET "/namespace/project.git/info/refs?service=git-upload-pack" for 192.168.122.20 at 2016-09-23 15:33:27 +0200
Processing by Projects::GitHttpController#info_refs as */*
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"namespace", "project_id"=>"project.git"}
Started POST "/ci/api/v1/builds/register.json" for 192.168.122.20 at 2016-09-23 15:33:27 +0200
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 126ms (Views: 0.8ms | ActiveRecord: 7.3ms)
Started GET "/namespace/project.git/info/refs?service=git-upload-pack" for 192.168.122.20 at 2016-09-23 15:33:27 +0200
Processing by Projects::GitHttpController#info_refs as */*
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"namespace", "project_id"=>"project.git"}
Completed 500 Internal Server Error in 238ms (ActiveRecord: 12.0ms)
JWT::DecodeError (Nil JSON web token):
lib/gitlab/workhorse.rb:120:in `verify_api_request!'
app/controllers/projects/git_http_client_controller.rb:154:in`verify_workhorse_api!'
lib/gitlab/request_profiler/middleware.rb:15:in `call'
lib/gitlab/middleware/go.rb:16:in `call'
Started POST "/ci/api/v1/builds/register.json" for 192.168.122.20 at 2016-09-23 15:33:28 +0200 In the same time we have the following error in the gitlab-workhorse logs (and similar one in the nginx logs) :
16:38:54.28695888 +0200 EET] "GET /namespace/project.git/info/refs?service=git-receive-pack HTTP/1.1" 500 2408 "" "git/2.7.2.windows.1" 0.275119
There are couple of other issues that mention similar problems. For ease of use, here are the links:
https://gitlab.com/gitlab-org/gitlab-ce/issues/22102 https://gitlab.com/gitlab-org/gitlab-ce/issues/22527
Activity
Mentioned in issue #22102 (closed)
Mentioned in issue #22527 (closed)
Mentioned in issue #22557 (closed)
@kmeh Are there any other related entries in the gitlab-workhorse logs?
@markglenfletcher - sorry that's the only entry @zehome - I've restarted the VM itself several times, so this can't be the problem.
thanks guys. I couldn't allow more downtime, so I've reverted to an back, which had some missing data, but it was better solution. The strange thing is that I applied the same updates to the backup, the only difference is that upgrade was from 8.8 to 8.12 instead of 8.11 to 8.12. I guess the reason for this whole thing is miss configuration on my part.
As far as I get it, there should be a difference if I update from 8.8 to 8.12 or from 8.11 to 8.12 - it's the same migrations, same data, same files. For me the issues is resolved. Should I close it, since I can't reproduce it?
