Backups fail occasionally with "tar: ./objects: file changed as we read it" error

Summary

Backups fail occasionally with "tar: ./objects: file changed as we read it" error which is probably caused by taking the backup with tar instead of using git bundle.

Steps to reproduce

1. take a backup: gitlab-rake gitlab:backup:create
2. perform some operation on the git repository
3. observe the backup failing

(This is probably a race condition due to using tar instead of git bundle so this does not reproduce all the time.)

Expected behavior

Backups not to fail

Relevant logs and/or screenshots

Backup failed
[FAILED]
failed: tar -cf /var/opt/gitlab/backups/repositories/path/to.bundle -C /var/opt/gitlab/git-data/repositories/path/to.git .
tar: ./objects: file changed as we read it

Output of checks

Results of GitLab Application Check

All tests passed.

Results of GitLab Environment Info

omnibus-gitlab CE

System information
System:         Debian 7.9
Current User:   git
Using RVM:      no
Ruby Version:   2.1.7p400
Gem Version:    2.2.5
Bundler Version:1.10.6
Rake Version:   10.4.2
Sidekiq Version:3.3.0

GitLab information
Version:        8.2.3
Revision:       19daba8
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.XX
HTTP Clone URL: https://gitlab.XXX.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.XXX:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        2.6.8
Repositories:   /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab

Possible fixes

I think this is a regression due to the change where backing up was changed to use tar instead of git bundle.

This was done in https://gitlab.com/gitlab-org/gitlab-ce/commit/d411a9e4d8063fdc9b6d0f74cad7345245a1fb0b so that git-annex files would be properly backed up.

Discussion on why backing up a git repository with tar is not necessarily a good idea: http://marc.info/?l=git&m=136422341014631&w=2

Maybe this could be fixed by locking the repository (i.e., preventing commits/other operations) when taking backups?

I am currently running version 8.1.4 (Omniauth installed). I am getting this same failure everyday now. The files that are changing for me are objects/pack/tmp_pack_*. I believe these are files created by git gc (garbage collection) so I am not sure that locking the repository will help. I have some large repos so gc can take a long time to run.

I just had this happen again, so it's not occurring that frequently for me.

According to tar documentation [1] exit code 1 is not fatal but indicates that some files differ. The tar snapshot is probably in an inconsistent state anyway if the object has changed during taring so considering exit 1 non-fatal is most likely a no-go.

[1] https://www.gnu.org/software/tar/manual/html_section/tar_19.html

@vsizov @marin @jacobvosmaer-gitlab Any ideas what we can do here. I have a customer that sees this daily because their instance is so busy. It's becoming a critical issue. I appreciate your input.

I can think of a few options:

1. Lock writes to the repository somehow (preventing git gc and pushes)
2. Clone each repo to another directory before backup

I started seeing this error as well and researched why they changed to using tar. Large file support was added to gitlab (ee I believe) and the "normal" git tools that used to be used to take a repo image do not include the large files. Thus they needed to backup the repo location using tar. One solution I have considered is to rsync the repo to be backed up to another location, then rsync again and again until no changes are detected between tow consecutive rsyncs. then you would use tar to backup the repo. I would like an option to keep the rsync'd repos as the base for the next backup. Yes this uses more space but it also improves backup times which are getting longer and longer. The rsync area can also act as 1 of the backups in the set. There are also opportunities to have rsyncs run throughout the day so that at backup time there is less to copy. As a repo is changed then it could be flagged as needing an rsync and put on a queue to be done when system is not so busy. You could also add parallel rsync processing if your I/O load can handle multiple rsyncs.

@gary.burchett yes we switched from git bundle to tar because of git-annex.

@stanhu I think git bundle is the moral equivalent of 'clone first' with the added benefit of skipping the intermediate directory. And I don't think git clone respects git-annex data.

We could try a dual approach: git bundle for Git data, and tar for the annex/ and custom_hooks/ directories.

Added backup-restore label

mentioned in issue #19130 (closed)

Added ~313020 label

@stanhu @jacobvosmaer-gitlab Any ideas on the best path forward? We should move quickly to make sure backups are solid. This is impacting a large customer.

I like @jacobvosmaer-gitlab's idea. Most people won't be using the annex/ dir, and the amount of data in custom_hooks is small.

OK then we know what to do. Who is going to do it? cc @DouweM

@DouweM We should do this as soon as possible - if not 8.11, then definitely 8.12. Having bad backups is bad times.

Added ~480950 label

Milestone changed to %8.11

Added ~167032 label

Added ~12980 label

@dblessing I've put it for 8.11. I'll play a bit with priorities so this floats to the top.

Added ~307500 label

@tiagonbotelho Can you have a look here?

Reassigned to @tiagonbotelho

mentioned in merge request !5814 (merged)

mentioned in issue #21101 (moved)

Milestone changed to %8.12

Hello GitLab team. How is this going?

On my CentOS 7 environment, this still occurs with the following setting:

/etc/gitlab/gitlab.rb

gitlab_rails['backup_path'] =  "<WebDAV(davfs2) mounted path>"

But after changing the setting to a local path, this problem does not occur.

gitlab_rails['backup_path'] =  "/var/opt/gitlab/backups"

My /etc/fstab is:

http://<WebDAV server>/dir /mnt/to/dir davfs rw,user,exec,uid=myuser,gid=myuser,_netdev 0 0

Workaround

Currently, I temporarily changed backup_path to a local path and gitlab-ctl reconfigure, and run the following commands with git user.

1. gitlab-rake gitlab:backup:create
2. rsync -av <local path>/ <real backup path>/

Milestone changed to %8.13

@tyru we are working on an improvement in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5814

@stanhu @jacobvosmaer-gitlab I already made so that git data is bundled and annex and custom_hooks are tarred I still need to block updates until the backup is made right?

Mentioned in issue gitlab-com/infrastructure#415 (closed)

Mentioned in merge request !5814 (merged)

Mentioned in issue #22689 (closed)

Milestone changed to %8.14

Status changed to closed by merge request !5814 (merged)

mentioned in issue #26881 (closed)

A customer encounters this error when pipeline schedule creates an artifact during backup.

reopened

removed assignee

Test Note

62ee978f

moved to gitlab#4073

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue