Squashing MRs doesn't work on GitLab.com

Reported at https://gitlab.com/gitlab-com/support-forum/issues/1532.

I tried with https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1192, and it works for me locally. I found some error logs in Kibana: https://log.gitlap.com/goto/e59a61283cfe1081af1d3a33389a7ff7

usage: git diff [--no-index] <path> <path>

This is what git diff says when it's not in a repo, which is weird. Additionally, some fail at the git commit step, but I don't see any stderr / stdout in the logs: https://log.gitlap.com/goto/f270a55a0b518ec98dcb8dee37b0c643

@stanhu I'm totally on this, as it works for me locally - but there are going to be so many differences in the environments that it's hard to say.

The basic steps squashing does are:

$ cd /path/to/repo/storage
$ git worktree add $squash_path $target_branch --detach
$ cd $squash_path # this should now be a worktree
$ git diff --binary $start_sha...$head_sha | git apply --index
$ git commit -C $head_sha

Do you have any ideas? If not, tomorrow I'm going to try with an omnibus package and see if anything's weird, or see if staging's up and I can poke around there.

mentioned in issue gitlab-com/support-forum#1532 (closed)

My best guess is that there is an environment difference with omnibus. I'd definitely test there first.

assigned to @smcgivern

@smcgivern git version?

@marin I've tried it on a VM with this package: https://packages.gitlab.com/gitlab/unstable/packages/ubuntu/xenial/gitlab-ee_8.17.0-rc1.ee.0_amd64.deb (Edit: and squashing worked )

That VM has git 2.7.4. What does gitlab.com have?

The package has git 2.10.2. Package does not use system git.

Right, thanks.

$ /opt/gitlab/embedded/bin/git --version
git version 2.10.2

I could really do with a working staging environment

This works on staging: smcgivern/squash-test!1 (closed)

But fails on production: https://gitlab.com/smcgivern/squash-test/merge_requests/1

It looks like kibana is a bit behind, so I'll have to wait for this to show up in there

Just chiming in to say I'm running into this on .com and a few people have reached out to me about it too.

We found the problem!

Squashing uses a worktree, which means that instead of cloning the bare repo and then pushing back to that remote, it creates a working copy from the bare repo and when we commit, commits happen directly in the bare repo.

When we ran it manually on production, we noticed that there was always an error immediately afterwards in the logs: git-annex: First run: git-annex init. I might have actually seen this in the logs above, but disregarded it as irrelevant, which was a mistake.

This still didn't make much sense, though, because git annex is enabled on staging, too, and should have nothing to do with commits - only pushes. We eventually realised that it was coming from a pre-commit hook, and then it took us a little longer to notice that ... there shouldn't be a pre-commit hook!

Production has this, on all workers, last modified somewhere between May and August 2016:

#!/bin/sh
# automatically configured by git-annex
git annex pre-commit .

Staging doesn't have it, and dev doesn't have it. It's also useless, because until we started using worktrees, we never committed on the server, just pushed! (As described above.)

We should move /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-commit to /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-commit.bak on all production workers for now, and that should have squash working - or at least, not completely broken!

Fixed on GitLab.com thanks to @MrChrisW!

Any new issues with squashing should be reported in the EE issue tracker and @Mention me

closed

This still existed on worker18; @omame renamed it there, too!

https://log.gitlap.com/goto/e0da4522ee6e2f42ff8e17fad6afe5e3

@smcgivern FWIW I tried to squash & merge https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9359 but it wasn't doing anything, so I ended up merging without squash.

@rymai thanks, that's interesting! It seems you tried at around 09:19 and 09:26 UTC. For the second one, I found this in the logs: ERROR -> Squash task canceled: Another squash is already in progress, but nothing for the first, not even a MergeWorker

Does that sound right to you?

@smcgivern Correct, I first tried it, then I saw that it was "doing nothing" (the merge button was greyed out and the spinner was spinning but nothing else), so I reloaded the page and tried again...

OK, so the problem with these appears to have been two things.

1. The pre-commit hook is added by git annex, which is annoying. We need to skip that with --no-verify. (To test this out, I ran ssh git@gitlab.com git-annex-shell configlist smcgivern/gitlab-ee.git locally, and this created the hook on worker13.

2. Creating worktrees for larger repos (like GitLab EE) is really slow. In a terminal, it took over two minutes to check out all the files. In a Ruby process, our git process got stuck! @ahmadsherif and I don't know why, but it seemed to stop doing anything other than printing the percentage of files checked out when we used strace.

   To fix this, and make squashes faster in general, we should set up sparse checkout as described by @chriscool in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1024#note_22876548.

1 also explains why the hook doesn't exist in staging - nobody uses git annex there, so it doesn't get created