Repository#fetch_geo_mirror quietly fails if the repository is broken

ZD: https://gitlab.zendesk.com/agent/tickets/71280

Somehow one of the customer's repositories got into a bad state. We ran GIT_TRACE=1 git fetch geo manually, and it didn't appear to do anything.

Then @MrChrisW had the good idea of trying to clone the bare repo, and this is what we got:

Because this repo was in a broken state, Geo was unable to update.

How did the repo wind up in this state? Possible reasons: bad initial rsync, aborted bad git fetch?

We need to:

1. Check the return value of Repository#fetch_geo_mirror.
2. Log an error to a specific place. This needs to be saved in a place where it's easy to look. Ideally also we can send this to the primary.
3. Flag this repo as unhealthy.
4. Design some recovery mechanism; consider rewiping and recloning if necesssary

added repository label

added ~1052112 customer labels

I was able to reproduce the error message in the screenshot by taking a valid repo and then:

1. Create commit A
2. Create commit B

In .git/objects, remove commit A.

$ git log
error: Could not read 4bc0184418ebeb75ea008778f6258088c948c997
fatal: Failed to traverse parents of commit df8e5c01cb652553223abcc9ad6b93be8a2bc094

I tarred this repository up to the primary. The secondary backfilled this repo and created a repo. A manual fetch resulted in this:

# sudo -u git git fetch geo
remote: error: Could not read 4bc0184418ebeb75ea008778f6258088c948c997
remote: fatal: Failed to traverse parents of commit df8e5c01cb652553223abcc9ad6b93be8a2bc094
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: aborting due to possible repository corruption on the remote side.
fatal: protocol error: bad pack header

@MrChrisW As I recall, in the customer's case, we got nothing of the sort. What's going on there?

@stanhu Thanks for reproducing! Is it possible git failed to fetch the remote for some reason (network). I would expect to still see an error of some kind (especially with GIT_TRACE enabled). The customer is still facing the issue, so I'll loop back and reproduce again.

I guess the difference in my case is that the primary was corrupted. I think the next step is to corrupt the secondary and keep the primary intact.

changed milestone to %9.4

I believe this is done now?

Geo::RepositorySyncService logs to geo.log if the git fetch fails for any reason.

We still don't have a recovery mechanism but we have https://gitlab.com/gitlab-org/gitlab-ee/issues/3119 to track that.

closed