backup fails running on ceph filesystem

I run gitlab from a ceph mounted filesystem. It runs from one host, i use ceph for stability only. Gitlab is stopped before i run backup, but it always fails with tar error: file changed as we read it. While i googled how to fix this i founded a tar parameter what may can help: --ignore-failed-read.

Sample:

juno gitlab # LC_ALL=C sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
Dumping database ...
Dumping PostgreSQL database gitlab_master ... [DONE]
done
Dumping repositories ...
 * erp-develop/i3sys-erp ... [DONE]
 * erp-develop/i3sys-erp.wiki ...  [SKIPPED]
 * voip-develop/kamailio-ws ... [DONE]
 * voip-develop/kamailio-ws.wiki ...  [SKIPPED]
 * voip-develop/kamailio-asterisk-transcoder ... [DONE]
 * voip-develop/kamailio-asterisk-transcoder.wiki ...  [SKIPPED]
 * server/i3-sip0 ... [DONE]
 * server/i3-sip0.wiki ...  [SKIPPED]
 * erp-develop/i3sys_addons_kabeltv ... [DONE]
 * erp-develop/i3sys_addons_kabeltv.wiki ...  [SKIPPED]
 * voip-develop/jssip ... [DONE]
 * voip-develop/jssip.wiki ...  [SKIPPED]
done
Dumping uploads ...
done
Creating backup archive: 1429825745_gitlab_backup.tar ... tar: repositories/erp-develop: file changed as we read it
tar: repositories: file changed as we read it
tar: uploads: file changed as we read it
creating archive 1429825745_gitlab_backup.tar failed
Backup failed

Status changed to closed

Status changed to reopened

Is this a ceph/tar problem?

Why is this a GitLab issue?

I am also running into this issue on a mounted CIFS drive:

Dumping uploads ...
done
Creating backup archive: 1435930409_gitlab_backup.tar ...tar: repositories: file changed as we read it
creating archive 1435930409_gitlab_backup.tar failed
Backup failed

This error message sometime alternates with the following error message:

Dumping uploads ...
done
Creating backup archive: 1436063512_gitlab_backup.tar ... creating archive 1436063512_gitlab_backup.tar failed

This is a gitlab issue, because i use that software. If gitlab use a creepy software for a function, that is gitlab's problem.

Got the same issue. Did you found any solutions?

How to add the --ignore-failed-read to the git backup script?

No. But in the mean time i moved to omnibus package on ubuntu, and now for an upgrade i am not forced to do backup.

Added ~65372 label

Removed ~65372 label

mentioned in issue gitlab-com/operations#1 (closed)

Added backup-restore label

/cc @pcarranza @jnijhof Has this been fixed?

Added ~65372 label

/cc @northrup

Is this still a problem? Or can this issue be closed down?

Closing this issue down in accordance with our Issue Triage Policies. Please reopen the issue if the you feel that the issue is still relevant. Thanks!

closed

mentioned in merge request !14785