Can't launch due to permission issues, and update-permissions aborts due to failure and doesn't fix it

I tried to launch gitlab recently after copying stuff back to the server and made some mistake with the permissions during copying.

It aborts due to wrong permissions on:

- /var/opt/gitlab/git-data/repositories
- /var/opt/gitlab/redis/redis.socket

.. and possibly more.

However, update-permissions is unable to fix it:

root@Ubuntu-1604-xenial-64-minimal /srv/gitlab # docker-compose run gitlab update-permissions
+ id -u git
998
+ chown -R git:git /var/opt/gitlab/.ssh
+ chown -R git:git /var/opt/gitlab/.gitconfig
+ chown -R git:git /var/opt/gitlab/git-data
+ chown -R git:git /var/opt/gitlab/gitlab-ci/builds
+ chown -R git:git /var/opt/gitlab/gitlab-rails
+ chown -R git:git /var/opt/gitlab/gitlab-shell
+ chown -R git:gitlab-www /var/opt/gitlab/gitlab-workhorse
+ chown git /var/opt/gitlab/gitlab-workhorse
+ chown git /var/log/gitlab/gitlab-rails
+ chown git /var/log/gitlab/gitlab-shell
+ chown git /var/log/gitlab/sidekiq
+ chown git /var/log/gitlab/unicorn
+ chown gitlab-psql /var/log/gitlab/postgresql
+ chown gitlab-redis /var/log/gitlab/redis
+ chown -R git:git /var/log/gitlab/gitlab-rails/application.log /var/log/gitlab/gitlab-rails/githost.log /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-1477143591-1168.log /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-1477642396-1185.log /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-1478887087-1288.log /var/log/gitlab/gitlab-rails/production.log /var/log/gitlab/gitlab-rails/repocheck.log /var/log/gitlab/gitlab-rails/sidekiq.log
+ chown -R git:git /var/log/gitlab/gitlab-shell/gitlab-shell.log
+ chown -R git:git /var/log/gitlab/unicorn/unicorn_stderr.log /var/log/gitlab/unicorn/unicorn_stdout.log
+ id -u gitlab-www
999
+ chown -R gitlab-www:gitlab-www /var/opt/gitlab/nginx/client_body_temp /var/opt/gitlab/nginx/fastcgi_temp /var/opt/gitlab/nginx/proxy_temp /var/opt/gitlab/nginx/scgi_temp /var/opt/gitlab/nginx/uwsgi_temp
+ chown -R gitlab-www:gitlab-www /var/opt/gitlab/nginx/proxy_cache
+ id -u gitlab-psql
996
+ chown -R gitlab-psql:gitlab-psql /var/opt/gitlab/postgresql
+ id -u gitlab-redis
997
+ chown gitlab-redis:gitlab-redis /var/opt/gitlab/redis
+ id -u registry
993
+ chown -R registry:registry /var/opt/gitlab/gitlab-rails/shared/registry
chown: cannot access '/var/opt/gitlab/gitlab-rails/shared/registry': No such file or directory
root@Ubuntu-1604-xenial-64-minimal /srv/gitlab #

Since this leaves me with an unlaunchable container, some recommended instructions on how to fix it would be neat! (especially since the error regarding /var/opt/gitlab/redis/redis.socket doesn't actually say what the permissions SHOULD be)

Just for what it's worth, I managed to fix it by manually changing some permissions. Therefore, the script update-permissions is definitely missing some of the required permission fixes for some situations.

I was migrating a gitlab server today (running through docker). I, too, hit this issue.

I managed to resolve it with chmod 2770 /srv/gitlab/data/git-data/repositories

I also removed the old redis data before starting rm /srv/gitlab/data/redis/dump.rdb

added gitlab-ce~1672339 label

moved from gitlab-ce#24432

added customer label

Hey @marin support would love if we could prioritize this for next release. This is our scheduling request. :)

@lbot 9.2 is already scheduled, maybe 9.3. /cc @regisF

changed milestone to %9.3

added SP3 label

This customers is eagerly awaiting this https://gitlab.zendesk.com/agent/tickets/68055

added For Scheduling label

removed milestone

Hey @marin, can we prioritize this one? This is a scheduling request from the support team

in the update-permissions script, we need to ignore the registry directory if it doesn't exist.

The error can be ignored currently, if you aren't running the gitlab registry.

added Scheduled and removed For Scheduling labels

changed milestone to %9.4

After update-permissions is run, I need to run chgrp git /var/opt/gitlab/redis inside the docker container to fix it.

/cc @WarheadsSE for your attention

assigned to @WarheadsSE

I've looked into this, and while fixing this script, I have run into an issue familiar to others at this time: https://gitlab.com/gitlab-org/gitlab-ce/issues/32895 .. we may want to move that issue here, as it is about the container itself.

mentioned in commit e75d354b

mentioned in merge request !1688 (merged)

I can confirm, on a fresh container with empty mounts, I am seeing what users are reporting here.

1. the error message, which I have fixed and MR'ing https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1688
2. complete failure to connect to Redis by all services. Errno::EACCES - Permission denied - connect(2) for /var/opt/gitlab/redis/redis.socket which I am looking into.

Alright, so I've narrowed down that the update-permissions script changes the permissions on the contained /var/opt/gitlab/redis to be gitlab-redis:gitlab-redis, which prevents the git user from accessing the contained redis.socket. This would then cause problems in the container, however, gitlab-ctl reconfigure that is called as a part of docker restart process should be fixing this.

To regard the original issue reported:

• /var/opt/gitlab/git-data/repositories should be handled by the chown -R git:git /var/opt/gitlab/git-data that is a part of the update-permissions call
• /var/opt/gitlab/redis/redis.socket should not be re-occurring after following the steps for fixing permissions (docker exec -ti gitlab update-permissions; docker restart gitlab)

mentioned in merge request !1706 (merged)

closed via merge request !1688 (merged)

closed via commit cb7d860a

closed via commit e75d354b

mentioned in commit cb7d860a