Backup fails if data changes during backup

Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/5613, which only addressed the issue with repository data. Now we're seeing this fairly often in GitHost, especially with the registry data. If we're backing up the registry data when a new image is pushed, the backup fails. This will be the case with any backup data where we're doing a direct tar of the data, without copying it first.

What is the best solution?

• Copy the data for a given component first, then tar?
• See if we can do some crazy streaming where we copy and pipe to the tar at the same time? I'm not even sure if this is a thing, but in my head it seems plausible.

At any rate, this is a potentially large problem for customers and for us on GitHost. We need reliable unsupervised backups.

cc/ @jacobvosmaer-gitlab @stanhu because you were involved in the discussion on the related issue. Of note, to solve the other issue we reverted to using git bundle for all but annex data - so we just sort of sidestepped the issue in that case. Here we need a different solution.

added ~1071780 backup-restore labels

Copy the data for a given component first, then tar?

I think this is the most straightforward approach. The alternative is to do something like log all the files that changed and retry until they succeed.

mentioned in merge request !8728 (merged)

closed via merge request !8728 (merged)