'Forbidden' after git lfs push in 8.12
I'm running a dockerised omnibus gitlab, which sits behind a reverse-proxy. After upgrading to 8.12 I decided to see if LFS now works with SSH.
I fetched all LFS objects with git lfs fetch --all using good all HTTPS remote and then created a new project, to which I wanted to push using SSH remote url. Here's the result:
git push new-origin master
Git LFS: (0 of 98 files, 3 skipped) 16.87 MB / 16.87 MB, 85.35 KB skipped
batch response: Authorization error: https://gitlab.example.com/my-username/test-repo.git/info/lfs/objects/batch
Check that you have proper access to the repositoryGoing to gitlab.example.com now results a hard-core 403Forbidden message, which is a sign of that the IP is banned. Here's a chunk of the server logs:
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/my-username/test-repo.git/info/lfs/objects/batch" for 172.18.0.2 at 2016-09-23 13:19:05 +0000
Processing by Projects::LfsApiController#batch as JSON
Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"a0c5f6da36e8b309fbc88c9f45e5f98dcaa2467c5e92395187bcd5cba424190c", "size"=>7427}, !!!+ ABOUT 100 MORE!!! ]}}
Completed 200 OK in 516ms (Views: 20.5ms | ActiveRecord: 57.0ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
172.18.0.2 - my-username [23/Sep/2016:13:19:06 +0000] "POST /my-username/test-repo.git/info/lfs/objects/batch HTTP/1.1" 200 5965 "-" "git-lfs/1.4.1 (GitHub; linux amd64; go 1.7; git 2070e4c)"
==> /var/log/gitlab/gitlab-workhorse/current <==
2016-09-23_13:19:06.13682 gitlab.example.com @ - - [2016-09-23 13:19:05.571599867 +0000 UTC] "POST /my-username/test-repo.git/info/lfs/objects/batch HTTP/1.1" 200 38390 "" "git-lfs/1.4.1 (GitHub; linux amd64; go 1.7; git 2070e4c)" 0.563649
==> /var/log/gitlab/sshd/current <==
2016-09-23_13:19:06.86234 Accepted publickey for git from 91.103.19.58 port 39390 ssh2: RSA SHA256:VoP+WNPJmW9yAlKJ/dOz97VMe31uQ87BSaO18h96Low
==> /var/log/gitlab/gitlab-rails/production.log <==
Started PUT "/my-username/test-repo.git/gitlab-lfs/objects/46d535ad08d38eed4de3e97aac6c5ce7e98742845dbd5ae543f6e055db579d38/418310/authorize" for 172.18.0.2 at 2016-09-23 13:19:07 +0000
Processing by Projects::LfsStorageController#upload_authorize as HTML
Parameters: {"namespace_id"=>"my-username", "project_id"=>"test-repo.git", "oid"=>"46d535ad08d38eed4de3e97aac6c5ce7e98742845dbd5ae543f6e055db579d38", "size"=>"418310"}
Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-09-23 13:19:07 +0000
Completed 200 OK in 162ms (Views: 0.4ms | ActiveRecord: 9.4ms)
Started PUT "/my-username/test-repo.git/gitlab-lfs/objects/46d535ad08d38eed4de3e97aac6c5ce7e98742845dbd5ae543f6e055db579d38/418310" for 172.18.0.2 at 2016-09-23 13:19:07 +0000
Processing by Projects::LfsStorageController#upload_finalize as HTML
Parameters: {"namespace_id"=>"my-username", "project_id"=>"test-repo.git", "oid"=>"46d535ad08d38eed4de3e97aac6c5ce7e98742845dbd5ae543f6e055db579d38", "size"=>"418310"}
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
I, [2016-09-23T13:19:07.841159 #659] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.25901
I, [2016-09-23T13:19:07.841535 #659] INFO -- : gitlab-shell: Processing LFS authentication for user with key key-5.
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/api/v3/internal/lfs_authenticate" for 127.0.0.1 at 2016-09-23 13:19:07 +0000
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
I, [2016-09-23T13:19:07.899602 #659] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/lfs_authenticate 0.05484
==> /var/log/gitlab/gitlab-rails/production.log <==
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 234ms (Views: 22.6ms | ActiveRecord: 11.0ms)
==> /var/log/gitlab/gitlab-workhorse/current <==
2016-09-23_13:19:07.96383 gitlab.example.com @ - - [2016-09-23 13:19:07.498374467 +0000 UTC] "PUT /my-username/test-repo.git/gitlab-lfs/objects/46d535ad08d38eed4de3e97aac6c5ce7e98742845dbd5ae543f6e055db579d38/418310 HTTP/1.1" 401 26 "" "git-lfs/1.4.1 (GitHub; linux amd64; go 1.7; git 2070e4c)" 0.465219
==> /var/log/gitlab/nginx/gitlab_access.log <==
172.18.0.2 - my-username [23/Sep/2016:13:19:07 +0000] "PUT /my-username/test-repo.git/gitlab-lfs/objects/46d535ad08d38eed4de3e97aac6c5ce7e98742845dbd5ae543f6e055db579d38/418310 HTTP/1.1" 401 26 "-" "git-lfs/1.4.1 (GitHub; linux amd64; go 1.7; git 2070e4c)"
==> /var/log/gitlab/sshd/current <==
2016-09-23_13:19:07.97116 Received disconnect from 91.103.19.58 port 39390:11: disconnected by user
2016-09-23_13:19:07.97147 Disconnected from 91.103.19.58 port 39390
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/my-username/test-repo.git/info/lfs/objects/batch" for 172.18.0.2 at 2016-09-23 13:19:08 +0000
Processing by Projects::LfsApiController#batch as JSON
Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"423e7fc5c2a8b566ff01f6ef36d986c75b3d014d27ca8a81a214ef4d8c2b5ea7", "size"=>554}], "namespace_id"=>"my-username", "project_id"=>"test-repo.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"423e7fc5c2a8b566ff01f6ef36d986c75b3d014d27ca8a81a214ef4d8c2b5ea7", "size"=>554}]}}
Started PUT "/my-username/test-repo.git/gitlab-lfs/objects/e965867172bfe1b8bae8b41ab6325aecd3e611fabf2169f3df5766fe0ccfce86/700343/authorize" for 172.18.0.2 at 2016-09-23 13:19:08 +0000
Processing by Projects::LfsStorageController#upload_authorize as HTML
Parameters: {"namespace_id"=>"my-username", "project_id"=>"test-repo.git", "oid"=>"e965867172bfe1b8bae8b41ab6325aecd3e611fabf2169f3df5766fe0ccfce86", "size"=>"700343"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 281ms (Views: 0.8ms | ActiveRecord: 115.9ms)
...For some reason HTTPS auth does not turn on correctly from SSH. Because LFS is trying to do quite a few requests, gitlab bans me completely (in my particular case everyone is banned because the reverse-proxy does not override headers correctly, but that's a different topic).
What could be wrong here?