Backup broken after upgrade to 8.8.1

Summary

Running sudo gitlab-rake gitlab:backup:create failed after an upgrade from 8.7.6 to 8.8.1.

Relevant logs

Result of the backup command with --trace, sudo gitlab-rake gitlab:backup:create --trace

...
Dumping lfs objects ... 
done
** Invoke gitlab:backup:registry:create (first_time)
** Invoke environment 
** Execute gitlab:backup:registry:create
Dumping container registry images ... 
rake aborted!
Errno::ENOENT: No such file or directory @ realpath_rec - /var/opt/gitlab/gitlab-rails/shared/registry
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:9:in `realpath'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:9:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/registry.rb:6:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:185:in `new'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:185:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `block in execute'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `each'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `execute'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:172:in `invoke_with_call_chain'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:165:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `block in execute'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `each'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `execute'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:172:in `invoke_with_call_chain'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/task.rb:165:in `invoke'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:150:in `invoke_task'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `each'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `block in top_level'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:115:in `run_with_threads'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:100:in `top_level'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:78:in `block in run'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:176:in `standard_exception_handling'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/lib/rake/application.rb:75:in `run'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.5.0/bin/rake:33:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/rake:23:in `load'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/rake:23:in `<main>'
Tasks: TOP => gitlab:backup:registry:create

Output of checks

Results of GitLab Application Check

GitLab check sudo gitlab-rake gitlab:check SANITIZE=true complained about uploads directory:

Uploads directory setup correctly? ... no
  Try fixing it:
  sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \;
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \;
  For more information see:
  doc/install/installation.md in section "GitLab"
  Please fix the error above and rerun the checks.

Running the suggested fixed worked for the uploads directory problem and all tests pass, but the the backup command produces the same output.

Results of GitLab Environment Info

Scrubbed URLs:

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.1.8p440
Gem Version:	2.5.1
Bundler Version:1.10.6
Rake Version:	10.5.0
Sidekiq Version:4.1.2

GitLab information
Version:	8.8.1
Revision:	9b9e320
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://url
HTTP Clone URL:	https://url/some-group/some-project.git
SSH Clone URL:	git@url:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	2.7.2
Repositories:	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:		/opt/gitlab/embedded/bin/git

I second that, same issue here.

mentioned in issue #17751 (closed)

Same issue here.

Created an empty registry dir as workaround:

mkdir /var/opt/gitlab/gitlab-rails/shared/registry
chown git /var/opt/gitlab/gitlab-rails/shared/registry

(What registry dir should contains?)

Milestone changed to %8.8

Added regression label

@ayufan is looking at this

Reassigned to @ayufan

Confirmed.

A workaround is

sudo mkdir -p /var/opt/gitlab/gitlab-rails/shared/registry

What are the correct ownership/permissions for this registry dir?

Added ~19173 label

registry:git

I created the registry directory and this allowed me to run the backup command, but when I try:

$ sudo chown registry:git registry
chown: invalid user: ‘registry:git’

There is no registry user on the machine.

Had a different server with different projects and repos. That server is using LDAP for authentication. Upgrade worked there. The registry user and directory exist on the machine that worked.

Just realized my other GitLab server that is running LDAP for authentication is running 8.8.0, not 8.8.1. I'll upgrade now to see if this causes the backup issue.

mentioned in merge request !4263 (merged)

mentioned in issue #17519 (closed)

mentioned in issue #17780 (closed)

I has same problem in GitLab 8.8.1

Status changed to closed by merge request !4263 (merged)

mentioned in commit f64fb6dd

Mentioned in commit pfjason/gitlab-ce@f64fb6dd

Mentioned in commit pfjason/gitlab-ce@019a3c78