Empty repository in CI Job after upgrade to 9.0.0

More info: we are using kubernetes runner with gitlab-ci-multi-runner 9.0.0 (gitlab/gitlab-runner:alpine).

On Gitlab 8.17.4 with Gitlab CI Multi Runner 1.11.1 everything works ok.

@vrtak-cz Could you check if you can pull the repository with HTTP locally?

no it looks like there is problem with HTTP (HTTPS) clone

I also encountered this issue. I'm not sure exactly what fixed it, but these hard reconfigs/restarts seems to have resolved the issue. I also erased and reran the pipeline. On gitlab instance: gitlab-ctl reconfigure && gitlab-ctl restart && reboot On runner: reboot

@jfols Thanks for reporting this. @vrtak-cz could you confirm it to be working?

@zj Thanks to @jfols Did gitlab-ctl reconfigure && gitlab-ctl restart and it helped me. Build stopped to fail on this issue and continue working.

Thanks @jfols after reconfigure & restart gitlab it works (no reboot / runner restart/reboot needed)

While trying to run the reconfigure I'm getting the follow.

Recipe: gitlab::gitlab-shell
  * ruby_block[directory resource: /var/opt/gitlab/git-data] action run (skipped due to not_if)
  * ruby_block[directory resource: /var/opt/gitlab/git-data/repositories] action run

    ================================================================================
    Error executing action `run` on resource 'ruby_block[directory resource: /var/opt/gitlab/git-data/repositories]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Failed asserting that mode permissions on "/var/opt/gitlab/git-data/repositories" is 2770
    ---- Begin output of set -x && [ "$(stat --printf='%04a' $(readlink -f /var/opt/gitlab/git-data/repositories) | grep -o '....$')" = '2770' ] ----
    STDOUT:
    STDERR: ++ grep -o '....$'
    +++ readlink -f /var/opt/gitlab/git-data/repositories
    ++ stat --printf=%04a /var/opt/gitlab/git-data/repositories
    + '[' 0770 = 2770 ']'
    ---- End output of set -x && [ "$(stat --printf='%04a' $(readlink -f /var/opt/gitlab/git-data/repositories) | grep -o '....$')" = '2770' ] ----
    Ran set -x && [ "$(stat --printf='%04a' $(readlink -f /var/opt/gitlab/git-data/repositories) | grep -o '....$')" = '2770' ] returned 1

    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:124:in `validate_command'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:112:in `block in validate'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:111:in `each_index'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:111:in `validate'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:87:in `validate!'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb:35:in `block (3 levels) in from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb

     26:   ruby_block "directory resource: #{params[:path]}" do
     27:     block do
     28:       # Ensure the directory exists
     29:       storage_helper.ensure_directory_exists(params[:path])
     30:
     31:       # Ensure the permissions are set
     32:       storage_helper.ensure_permissions_set(params[:path])
     33:
     34:       # Error out if we have not achieved the target permissions
     35:       storage_helper.validate!(params[:path])
     36:     end
     37:     not_if { storage_helper.validate(params[:path]) }
     38:   end
     39: end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb:26:in `block in from_file'

    ruby_block("directory resource: /var/opt/gitlab/git-data/repositories") do
      params {:path=>"/var/opt/gitlab/git-data/repositories", :owner=>"git", :group=>nil, :mode=>"2770", :name=>"/var/opt/gitlab/git-data/repositories"}
      action [:run]
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      block_name "directory resource: /var/opt/gitlab/git-data/repositories"
      declared_type :ruby_block
      cookbook_name "gitlab"
      recipe_name "gitlab-shell"
      block #<Proc:0x000000058e4658@/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb:27>
      not_if { #code block }
    end

    Platform:
    ---------
    x86_64-linux


Running handlers:
Running handlers complete
Chef Client failed. 0 resources updated in 05 seconds

@mickfeech The output suggests you have the wrong permissions set for the directory containing repository data. But this seems unrelated to the current issue. Could you open a new one on the omnibus project?

@zj yeah. It appears someone was monkeying with permissions on the repo and had removed the sticky bit on the directory. Interesting that chef bombed trying to make the permissions correct, as it knew they were wrong.

This issue also breaks git push via HTTPS, gitlab-ctl reconfigure && gitlab-ctl restart fixed it for us.

@SneakyMax Are you able to check in gitlab-ctl reconfigure's output what changes when it's applied? Maybe this would show us what is the root cause of the issue since currently we only know how to solve it, not why we need to solve it 😉

It seems gitlab-ctl restart solved the problem for this user: https://gitlab.com/gitlab-org/gitlab-ee/issues/1994#note_26010114

Is it possible that Gitaly or some other process wasn't running?

@stanhu This should be easy to test, I can check this.

@stanhu It looks that you was right.

What I've done to reproduce the issue:

I've installed GitLab CE 8.17.4-ce.0 from ubuntu package

After installing it I've executed gitlab-ctl reconfigure:

root@ubuntu:~# gitlab-ctl reconfigure
(...)

Running handlers:
Running handlers complete
Chef Client finished, 235/339 resources updated in 01 minutes 50 seconds
gitlab Reconfigured!

I've checked what gitlab-ctl status shows at this moment:

root@ubuntu:~# gitlab-ctl status
run: gitlab-workhorse: (pid 10091) 54s; run: log: (pid 9983) 85s
run: logrotate: (pid 10009) 77s; run: log: (pid 10008) 77s
run: nginx: (pid 9994) 83s; run: log: (pid 9993) 83s
run: postgresql: (pid 9837) 144s; run: log: (pid 9836) 144s
run: redis: (pid 9749) 150s; run: log: (pid 9748) 150s
run: sidekiq: (pid 9974) 86s; run: log: (pid 9973) 86s
run: unicorn: (pid 9943) 92s; run: log: (pid 9942) 92s

All services were running properly at this moment.

I've updated GitLab CE to 9.0.0-ce.0, this automatically started gitlab-ctl reconfigured:

root@ubuntu:~# apt-get install gitlab-ce=9.0.0-ce.0
(...)

Running handlers:
Running handlers complete
Chef Client finished, 2/391 resources updated in 09 seconds
gitlab Reconfigured!
Restarting previously running GitLab services
ok: run: gitlab-workhorse: (pid 19276) 0s
ok: run: logrotate: (pid 19284) 0s
ok: run: nginx: (pid 19290) 1s
ok: run: postgresql: (pid 18797) 15s
ok: run: redis: (pid 18805) 15s
ok: run: sidekiq: (pid 19300) 0s
ok: run: unicorn: (pid 19304) 1s

Upgrade complete! If your GitLab server is misbehaving try running

   sudo gitlab-ctl restart

before anything else. If you need to roll back to the previous version you can
use the database backup made during the upgrade (scroll up for the filename).

Note the line Restarting previously running GitLab services!

I've checked what services are running with gitlab-ctl status:

root@ubuntu:~# gitlab-ctl status
down: gitaly: 1474s, normally up; run: log: (pid 18486) 1518s
down: gitlab-monitor: 1473s, normally up; run: log: (pid 18557) 1508s
run: gitlab-workhorse: (pid 19276) 1454s; run: log: (pid 9983) 2206s
run: logrotate: (pid 19284) 1454s; run: log: (pid 10008) 2198s
run: nginx: (pid 19290) 1454s; run: log: (pid 9993) 2204s
down: node-exporter: 1471s, normally up; run: log: (pid 18532) 1511s
down: postgres-exporter: 1471s, normally up; run: log: (pid 18548) 1509s
run: postgresql: (pid 18797) 1468s; run: log: (pid 9836) 2265s
down: prometheus: 1471s, normally up; run: log: (pid 18520) 1513s
run: redis: (pid 18805) 1469s; run: log: (pid 9748) 2272s
down: redis-exporter: 1470s, normally up; run: log: (pid 18540) 1511s
run: sidekiq: (pid 19300) 1454s; run: log: (pid 9973) 2208s
run: unicorn: (pid 19304) 1454s; run: log: (pid 9942) 2214s

Gitaly service (and also some others) is down!

I've executed gitlab-ctl restart and double checked this with gitlab-ctl status:

root@ubuntu:~# gitlab-ctl restart
ok: run: gitaly: (pid 21003) 1s
ok: run: gitlab-monitor: (pid 21010) 0s
ok: run: gitlab-workhorse: (pid 21013) 0s
ok: run: logrotate: (pid 21024) 1s
ok: run: nginx: (pid 21030) 0s
ok: run: node-exporter: (pid 21038) 1s
ok: run: postgres-exporter: (pid 21045) 0s
ok: run: postgresql: (pid 21055) 1s
ok: run: prometheus: (pid 21063) 0s
ok: run: redis: (pid 21072) 0s
ok: run: redis-exporter: (pid 21077) 0s
ok: run: sidekiq: (pid 21084) 0s
ok: run: unicorn: (pid 21089) 0s

root@ubuntu:~# gitlab-ctl status
run: gitaly: (pid 21003) 16s; run: log: (pid 18486) 1544s
run: gitlab-monitor: (pid 21010) 15s; run: log: (pid 18557) 1534s
run: gitlab-workhorse: (pid 21013) 15s; run: log: (pid 9983) 2232s
run: logrotate: (pid 21024) 15s; run: log: (pid 10008) 2224s
run: nginx: (pid 21030) 14s; run: log: (pid 9993) 2230s
run: node-exporter: (pid 21038) 14s; run: log: (pid 18532) 1537s
run: postgres-exporter: (pid 21045) 13s; run: log: (pid 18548) 1535s
run: postgresql: (pid 21055) 13s; run: log: (pid 9836) 2291s
run: prometheus: (pid 21063) 12s; run: log: (pid 18520) 1538s
run: redis: (pid 21072) 12s; run: log: (pid 9748) 2297s
run: redis-exporter: (pid 21077) 11s; run: log: (pid 18540) 1536s
run: sidekiq: (pid 21084) 9s; run: log: (pid 9973) 2233s
run: unicorn: (pid 21100) 8s; run: log: (pid 9942) 2239s

I don't have knowledge why there is a policy to Restarting previously running GitLab services instead of something like Starting all required GitLab services (@marin @twk3 I think this is the question for you), but with such everyone who will just upgrade the package but not run gitlab-ctl restart (or start each missing service separately) will fall into this issue.

@tmaczukin Sorry I wasn't the one to run the command ☹️ just wanted to chime in that git push was also broken because of this.

@SneakyMax No problem. I suspected that something important is changed during gitlab-ctl reconfigure. And you was last person that reported the issue, two minutes before my request, so there was a chance that you still have the output of command on the screen 😉

mentioned in issue gitaly#153 (closed)

@tmaczukin Thanks for reproducing the problem in such great detail! It sounds like there are two problems:

Clones should fail when Gitaly is not available (gitaly#153 (closed))
Omnibus should start required processes after the reconfigure

I'll move this issue to the omnibus-gitlab tracker since it seems more applicable there.

moved from gitlab-ce#29855

Currently we are restarting only the processes that were previously running, as seen in code. We do this using a regex matcher against run in service statuses. What we can do is the reverse: Start everything that wasn't down already. That is inverse-regex match against down in service statuses.

@twk3 @stanhu Since the original code was written way before, I don't know about the problems of the proposed approach. I gave it a try and it seems to have worked with default options. I am yet to test with certain services disabled and all. Still, can you provide input on the proposal?

mentioned in issue gitlab-workhorse#121 (closed)

@balasankarc Nice work. That looks good to me. Based on the runit source code, there are a number of possible strings:

run
finish
down

I don't think we ever have seen finish, though.

@stanhu I know I haven't. @marin or @jacobvosmaer-gitlab can confirm.

Since it is easier to comment, I will open an MR.

mentioned in merge request !1427 (merged)

I think the idea in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2119#note_26031750 is great! As Stan points out, we do have to be careful how we 'invert' the regex. And yes, easier to judge once we see code.

mentioned in issue gitlab-qa#37

A customer hit this problem (https://gitlab.zendesk.com/agent/tickets/71109) and confused them to the point they rolled back. 9.0.1 returned an Error 500 as intended via https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/145, while 9.0.0 returned an empty repo during an HTTP clone.

I recommend that we try to fix this (e.g. via !1427 (merged)) ASAP.

/cc: @marin, @twk3

closed via merge request !1427 (merged)

mentioned in commit 42c9af27

mentioned in merge request !1466 (merged)

Empty repository in CI Job after upgrade to 9.0.0

Summary

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Results of GitLab application Check

Designs

Child items 0

Activity

Admin message

Admin message

Empty repository in CI Job after upgrade to 9.0.0

Summary

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Results of GitLab application Check

Activity