Docker Omnibus GitLab configuration fails on "Error executing action 'run' on resource ... /gitlab-keys check-permissions"

Summary

Docker Omnibus GitLab configuration fails on Error executing action 'run' on resource 'execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions]'

Steps to reproduce

Docker host machine is Linux CentOS (4.11.1-1.el7.elrepo.x86_64 #1 (closed) SMP Sun May 14 11:54:29 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux)

Here is my docker-compose.yml file which fails:

version: '3'
services:
    gitlab:
        image: 'gitlab/gitlab-ce:9.3.6-ce.0'
        restart: always
        privileged: true
        environment:
            GITLAB_OMNIBUS_CONFIG: |
              external_url 'http://127.0.0.1/gitlab'
              gitlab_rails['gitlab_shell_ssh_port'] = 2233
              gitlab_rails['rails_relative_url_root'] = "/gitlab"
        ports:
            - '8033:80'
            - '2233:22'
            - '4433:443'
        volumes:
            - './config:/etc/gitlab'
            - './logs:/var/log/gitlab'

What is the current bug behavior?

GitLab Omnibus fails with error STDERR: /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:703:in 'initialize': Permission denied @ rb_sysopen - /var/log/gitlab/gitlab-shell/gitlab-shell.log (Errno::EACCES) (see full output bellow).

When I execute recommended commands Command docker exec -it gitlab update-permissions more accurately its "compose" version docker-compose exec gitlab update-permissions (see full output bellow) and docker-compose restart, The 'Permission denied' Error still persist.

What is the expected correct behavior?

GitLab Omnibus configuration should run without errors.

Relevant logs and/or screenshots

GitLab Omnibus fails with this error report:

gitlab_1  | Thank you for using GitLab Docker Image!
gitlab_1  | Current version: gitlab-ce=9.3.6-ce.0
gitlab_1  |
gitlab_1  | Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
gitlab_1  | And restart this container to reload settings.
gitlab_1  | To do it use docker exec:
gitlab_1  |
gitlab_1  |   docker exec -it gitlab vim /etc/gitlab/gitlab.rb
gitlab_1  |   docker restart gitlab
gitlab_1  |
gitlab_1  | For a comprehensive list of configuration options please see the Omnibus GitLab readme
gitlab_1  | https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
gitlab_1  |
gitlab_1  | If this container fails to start due to permission problems try to fix it by executing:
gitlab_1  |
gitlab_1  |   docker exec -it gitlab update-permissions
gitlab_1  |   docker restart gitlab
gitlab_1  |
gitlab_1  | Preparing services...
gitlab_1  | Starting services...
gitlab_1  | Configuring GitLab package...
gitlab_1  | Configuring GitLab...
gitlab_1  |
gitlab_1  | ================================================================================
gitlab_1  | Error executing action `run` on resource 'execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions]'
gitlab_1  | ================================================================================
gitlab_1  |
gitlab_1  | Mixlib::ShellOut::ShellCommandFailed
gitlab_1  | ------------------------------------
gitlab_1  | Expected process to exit with [0], but received '1'
gitlab_1  | ---- Begin output of /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions ----
gitlab_1  | STDOUT:
gitlab_1  | STDERR: /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:703:in `initialize': Permission denied @ rb_sysopen - /var/log/gitlab/gitlab-shell/gitlab-shell.log (Errno::EACCES)
gitlab_1  |     from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:703:in `open'
gitlab_1  |     from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:703:in `open_logfile'
gitlab_1  |     from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:695:in `set_dev'
gitlab_1  |     from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:635:in `initialize'
gitlab_1  |     from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:353:in `new'
gitlab_1  |     from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:353:in `initialize'
gitlab_1  |     from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:15:in `new'
gitlab_1  |     from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:15:in `<top (required)>'
gitlab_1  |     from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_keys.rb:4:in `require_relative'
gitlab_1  |     from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_keys.rb:4:in `<top (required)>'
gitlab_1  |     from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
gitlab_1  |     from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
gitlab_1  |     from /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys:24:in `<main>'
gitlab_1  | ---- End output of /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions ----
gitlab_1  | Ran /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions returned 1
gitlab_1  |
gitlab_1  | Resource Declaration:
gitlab_1  | ---------------------
gitlab_1  | # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-shell.rb
gitlab_1  |
gitlab_1  | 115: execute "#{gitlab_shell_keys_check} check-permissions" do
gitlab_1  | 116:   user git_user
gitlab_1  | 117:   group git_group
gitlab_1  | 118: end
gitlab_1  | 119:
gitlab_1  |
gitlab_1  | Compiled Resource:
gitlab_1  | ------------------
gitlab_1  | # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-shell.rb:115:in `from_file'
gitlab_1  |
gitlab_1  | execute("/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions") do
gitlab_1  |   action [:run]
gitlab_1  |   retries 0
gitlab_1  |   retry_delay 2
gitlab_1  |   default_guard_interpreter :execute
gitlab_1  |   command "/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions"
gitlab_1  |   backup 5
gitlab_1  |   group "git"
gitlab_1  |   returns 0
gitlab_1  |   user "git"
gitlab_1  |   declared_type :execute
gitlab_1  |   cookbook_name "gitlab"
gitlab_1  |   recipe_name "gitlab-shell"
gitlab_1  | end
gitlab_1  |
gitlab_1  | Platform:
gitlab_1  | ---------
gitlab_1  | x86_64-linux

Command docker exec -it gitlab update-permissions, more accurately its "compose" version docker-compose exec gitlab update-permissions has this output:

+ id -u git
998
+ chown -R git:git /var/opt/gitlab/.ssh
+ chown -R git:git /var/opt/gitlab/.gitconfig
+ chown -R git:git /var/opt/gitlab/git-data
+ chown -R git:git /var/opt/gitlab/gitlab-ci/builds
chown: cannot access '/var/opt/gitlab/gitlab-ci/builds': No such file or directory
+ chown -R git:git /var/opt/gitlab/gitlab-rails
chown: cannot access '/var/opt/gitlab/gitlab-rails': No such file or directory
+ chown -R git:git /var/opt/gitlab/gitlab-shell
+ chown -R git:gitlab-www /var/opt/gitlab/gitlab-workhorse
chown: cannot access '/var/opt/gitlab/gitlab-workhorse': No such file or directory
+ chown git /var/opt/gitlab/gitlab-workhorse
chown: cannot access '/var/opt/gitlab/gitlab-workhorse': No such file or directory
+ chown git /var/log/gitlab/gitlab-rails
chown: cannot access '/var/log/gitlab/gitlab-rails': No such file or directory
+ chown git /var/log/gitlab/gitlab-shell
+ chown git /var/log/gitlab/sidekiq
chown: cannot access '/var/log/gitlab/sidekiq': No such file or directory
+ chown git /var/log/gitlab/unicorn
chown: cannot access '/var/log/gitlab/unicorn': No such file or directory
+ chown gitlab-psql /var/log/gitlab/postgresql
chown: cannot access '/var/log/gitlab/postgresql': No such file or directory
+ chown gitlab-redis /var/log/gitlab/redis
chown: cannot access '/var/log/gitlab/redis': No such file or directory
+ chown -R git:git '/var/log/gitlab/gitlab-rails/*.log'
chown: cannot access '/var/log/gitlab/gitlab-rails/*.log': No such file or directory
+ chown -R git:git '/var/log/gitlab/gitlab-shell/*.log'
chown: cannot access '/var/log/gitlab/gitlab-shell/*.log': No such file or directory
+ chown -R git:git '/var/log/gitlab/unicorn/*.log'
chown: cannot access '/var/log/gitlab/unicorn/*.log': No such file or directory
+ id -u gitlab-www
999
+ chown -R gitlab-www:gitlab-www '/var/opt/gitlab/nginx/*_temp'
chown: cannot access '/var/opt/gitlab/nginx/*_temp': No such file or directory
+ chown -R gitlab-www:gitlab-www '/var/opt/gitlab/nginx/*_cache'
chown: cannot access '/var/opt/gitlab/nginx/*_cache': No such file or directory
+ id -u gitlab-psql
996
+ chown -R gitlab-psql:gitlab-psql /var/opt/gitlab/postgresql
chown: cannot access '/var/opt/gitlab/postgresql': No such file or directory
+ id -u gitlab-redis
997
+ chown -R gitlab-redis:gitlab-redis /var/opt/gitlab/redis
chown: cannot access '/var/opt/gitlab/redis': No such file or directory
+ id -u registry
993
+ chown -R registry:registry /var/opt/gitlab/gitlab-rails/shared/registry
chown: cannot access '/var/opt/gitlab/gitlab-rails/shared/registry': No such file or directory

Results of GitLab environment info

$ docker-compose exec gitlab sudo gitlab-rake gitlab:env:info
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "exec: \"sudo\": executable file not found in $PATH"

Results of GitLab application Check

$ docker-compose exec gitlab sudo gitlab-rake gitlab:check SANITIZE=true
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "exec: \"sudo\": executable file not found in $PATH"

$ docker-compose exec gitlab sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "exec: \"sudo\": executable file not found in $PATH"

Possible fixes

I've googled all similar bugs and try to manually set permissions of /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys to -rwxrwxrwx, but it constantly returns to -rwxr-xr-x after restart:

$ docker-compose exec gitlab ls -la /opt/gitlab/embedded/service/gitlab-shell/bin/
total 21584
drwxr-xr-x 2 root root     4096 Jul 11 22:05 .
drwxr-xr-x 1 root root     4096 Jul 12 12:14 ..
-rwxr-xr-x 1 root root      894 Jul 11 20:04 authorized_keys
-rwxr-xr-x 1 root root      945 Jul 11 20:04 check
-rwxr-xr-x 1 root root      524 Jul 11 20:04 compile
-rwxr-xr-x 1 root root      829 Jul 11 20:04 create-hooks
-rwxr-xr-x 1 root root 11026578 Jul 11 20:04 gitaly-receive-pack
-rwxr-xr-x 1 root root 11026577 Jul 11 20:04 gitaly-upload-pack
-rwxr-xr-x 1 root root      741 Jul 11 20:04 gitlab-keys
-rwxr-xr-x 1 root root      882 Jul 11 20:04 gitlab-projects
-rwxr-xr-x 1 root root      588 Jul 11 20:04 gitlab-shell
-rwxr-xr-x 1 root root      816 Jul 11 20:04 install

I've successfully ran this docker-compose setup on local machines A) macOS Sierra 10.12.5 and B) Ubuntu 17.04, but I am not able to make it run on CentOS server.

Thank you in advance for any help.

/label bug

changed the description

OK the issue was caused by Linux ACL - this command fixed the issue setfacl -bR gitlab/

closed