Extend housekeeping feature with `git prune`

Today, while pushing changes to the GitLab CE repo, we saw a warning:

(...)
Total 18 (delta 17), reused 0 (delta 0)
error: The last gc run reported the following. Please correct the root cause and remove gc.log.
Automatic cleanup will not be performed until the file is removed.

warning: There are too many unreachable loose objects; run 'git prune' to remove them.

To git@gitlab.com:gitlab-org/gitlab-ce.git
(...)

It looks like git gc sometimes can't handle unreachable objects. gc is executed automatically by some git commands and can be executed by our housekeeping feature. But in such situation it probably won't help.

My proposition is to add to the housekeeping - as an option (e.g. a checkbox in settings page) - execution of git prune inside of the repository. As for now we can do it only by logging on the server by SSH and executing it manually.

This of course entail an update in the gitlab-shell.

/cc @JobV @DouweM @yorickpeterse

Obligatory mention of @jacobvosmaer in such threads.

We recently changed gitlab-shell to only do git gc --no-prune because of https://gitlab.com/gitlab-org/gitlab-ce/issues/13524

Maybe we should revert the gitlab-shell change. cc @grzesiek

On the other hand I don't know if https://gitlab.com/gitlab-org/gitlab-shell/commit/22b36aec8b8e0a49b4b33a11ac5f972e50cbd000 is even deployed yet on gitlab.com.

Nope, gitlab.com still does plain 'git gc'. @tmaczukin it is strange that you encountered that error then.

@tmaczukin also:

git@worker1:~/git-data/repositories/gitlab-org/gitlab-ce.git$ find . -name gc.log
git@worker1:~/git-data/repositories/gitlab-org/gitlab-ce.git$ date
Thu Mar 17 16:18:17 UTC 2016
git@worker1:~/git-data/repositories/gitlab-org/gitlab-ce.git$ ls -l objects/pack/
total 132016
-r--r--r-- 1 git git   6836516 Mar 17 15:57 pack-c24167452438d4eaf3e8c5f76fc21756a15f8c77.idx
-r--r--r-- 1 git git 128341589 Mar 17 16:02 pack-c24167452438d4eaf3e8c5f76fc21756a15f8c77.pack

It looks like 'auto gc' is active.

@jacobvosmaer I'm not sure if git gc without --no-prune does the same, what git prune does.

We didn't run housekeeping, @yorickpeterse juts executed git prune on the repository.

It looks like 'auto gc' is active.

I'm aware of this, but look at the message

error: The last gc run reported the following. Please correct the root cause and remove gc.log.
Automatic cleanup will not be performed until the file is removed.

warning: There are too many unreachable loose objects; run 'git prune' to remove them.

Automated gc run failed because of too many unreachable loose objects. And git also say to us: run 'git prune' to remove them. Not: run 'git gc'.

I think that running git prune is something that will be done really rarely, that's why I proposed to make this an option, not a normal housekeeping behavior.

Anyway - today execution of git prune fixed the problem. Maybe we should try to reproduce the problem and check if running only git gc would help. But I think it wouldn't (Please correct the root cause and remove gc.log).

@jacobvosmaer I've also read the #13524 (closed). Running git prune removed all unreachable objects... so it's exactly that, what shouldn't be done according to #13524 (closed). But - if I understand the warning properly - without this prune 'auto gc' wouldn't have any effect. Now it's more complicated than only allow to do git prune...

@tmaczukin git gc is a combination of git repack and git prune.

https://www.kernel.org/pub/software/scm/git/docs/git-gc.html

I'm not sure if git gc without --no-prune does the same, what git prune does.

I think I don't understand what you are saying. If it did the same, why would the --no-prune option even exist?

@tmaczukin I also don't understand what happened from the issue description. You say @yorickpeterse executed git prune? Where, on a gitlab.com server?

The error message you pasted talks about gc.log. That file is not in the gitlab-org/gitlab-ce.git bare repo on gitlab.com. So what is the error message talking about?

@jacobvosmaer I did the following:

1. SSH into worker1
2. cd into the gitlab-org/gitlab-ce directory
3. ran rm gc.log, this just contained the line " warning: There are too many unreachable loose objects; run 'git prune' to remove them."
4. ran git prune and prayed it didn't break things (which it thankfully didn't)

This sounds almost like a bug in git. A 'warning' (not an 'error'!) in gc.log was blocking all future gc runs?

Yes. And also last gc run didn't finished because of There are too many unreachable loose objects; run 'git prune' to remove them. That's why I assumed, that git gc and git prune are doing different things. Other way git gc should handled the unreachable objects (and the warning says to me "no, use prune instead").

And after reading the #13524 (closed) I see a problem:

• If we not prune unreachable objects, then git is preventing auto-gc and also is showing this warning.
• If we prune unreachable objects (to reenable auto-gc and remove the warning) we are dropping into the issue described in #13524 (closed).

@tmaczukin the funny thing is that by default, git gc will call git prune with options to only delete objects older than 2 weeks.

Maybe add --aggressive when explicitly called by user.

@ishitatsuyuki https://metalinguist.wordpress.com/2007/12/06/the-woes-of-git-gc-aggressive-and-how-git-deltas-work/ git gc --aggressive sounds like something we do not want.

This is my understanding now of what happens.

git gc by default includes git prune with a time based expiry rule: it prunes loose objects older than two weeks. In contrast, git gc --auto (which is run automatically after receiving a git push etc.) uses a count based expiry rule: if it counts too many loose objects it will warn (whine) about pruning. So if a repo accumulates a lot of loose objects in a period under two weeks, git gc --auto is bound to complain. A typical cause of loose objects is @yorickpeterse force pushing all the time .

Considering that we have now have our own version of git gc --auto I think we should disable it with git config. This is a manual step for source installations; for omnibus it can be automated easily.

We should also heed the warning and keep doing some sort of pruning. I think it would be best to go back to git gc (instead of the current git gc --no-prune) and have loose objects older than 2 weeks removed.

git config --global gc.auto 0 disables git gc --auto

So for this we need to change:

• gitlab-shell (remove --no-prune)
• gitlab-ce (git config in install guide, update guide)
• omnibus-gitlab (gitconfig)

Reassigned to @jacobvosmaer

mentioned in merge request !3572 (merged)

mentioned in merge request gitlab-shell!48 (merged)

mentioned in merge request omnibus-gitlab!719 (merged)

Milestone changed to 8.7

Status changed to closed by merge request omnibus-gitlab!719 (merged)

mentioned in commit omnibus-gitlab@8ed13f4b

Hmm this got closed accidentally.

Status changed to reopened

Status changed to closed by merge request !3572 (merged)

Again! :)

Status changed to reopened

The change in https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/48 needs to end up in gitlab 8.7.0 before we close this.

Status changed to closed by commit dblessing/omnibus-gitlab@8ed13f4b5a932839449b34893dfe9dfd0e99436e