Rebasing not working with Merge Requests

Users Can't rebase their Merge Requests. We click the rebase button but it fails.

Steps to reproduce:

1. Enable rebase with only fast forwarding for merge requests.
2. Go to a merge request for a branch that cannot be fast forwarded
3. Click rebase button
4. It will spin for a while, depends on size of repository
5. Eventually the rebase button turns back to green. Something failed in the above attempt.

More info here: https://www.youtube.com/watch?v=yJAfTZB1Hhs

ZD: https://gitlab.zendesk.com/agent/tickets/80294

cc// @victorwu

/cc @vsizov

added gitlab-ce~13921 label

Any update on this?

/cc @smcgivern

Possible duplicate of https://gitlab.com/gitlab-org/gitlab-ee/issues/1774

mentioned in issue #1774 (closed)

@markglenfletcher I thought #1774 (closed) was more about about how fails silently. The fix for that one is to expose some error message to the frontend... The fix for this one is, well, I'm not sure, I don't see any error so I'm not sure why the rebase is failing.

@david.grant this will be fixed by #1774 (closed) in the sense that it will show you an error

Can you create a public repo demonstrating the issue, please?

@smcgivern I created a repo here and it seems be easily reproducible there too

https://gitlab.com/david.grant/reproduce-issue/merge_requests/1

Can anyone look at the rebase logs for david.grant/reproduce-issue!1 (merged) and give any insight on what the problem is?

+1 on this issue, we're having to manually rebase+squash+merge to work around this

mentioned in issue #2830 (closed)

+1 Experiencing this on gitlab.com

@david.grant thanks for the example repo! Is there any reason it is private? I would like to invite a colleague to look, but I'm not sure if you made it private for a reason.

Anyway, in that case, it looks like the MR is in fact not in need of a rebase at all: the source branch is already fast-forwardable from the target branch. I can't reproduce this locally, so I'll try to figure out why it's happening on GitLab.com.

@smcgivern Sorry, I made it private by accident. It should be public now for all to see!

david.grant/reproduce-issue!1 (merged) does need a rebase. Here's my trying to do a fast-forward merge from that branch into master on the command line:

david@david-p50:~/git/reproduce-issue$ git co master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
david@david-p50:~/git/reproduce-issue$ git merge --ff-only branch-1 
fatal: Not possible to fast-forward, aborting.

And the tree as seen in gitk:

@david.grant ah, I think this is the source of the confusion. The source branch has updated, but the MR hasn't. So if you clone that repository now, and look at the graph, branch-1 is fast-forwardable. A workaround for this issue is to close and reopen the MR (like I just did) which triggers an update.

I'm looking into why this happens.

The problem is this: https://sentry.gitlap.com/gitlab/gitlabcom/issues/37256/

It looks like our changes to gitlab-shell and PostReceive don't work for rebasing. Using a worktree (https://gitlab.com/gitlab-org/gitlab-ee/issues/1604) would implicitly fix this, but I wonder if there's something we can do to fix this in a patch release without that refactor.

added regression label

changed milestone to %9.4

assigned to @vsizov

@smcgivern Thanks, my bad for not updating my local before posting that info... Ok, added another master commit, going to try rebasing again and doing the close/open trick.

@smcgivern @vsizov Yep, confirmed! Hitting rebase (this worked, I confirmed by pulling and I see the branch remote got updated/rebased), close MR, open MR, and then the UI shows rebase is done and I can now merge. Well that's great news that there's nothing really wrong with the rebase here, just state not being updated correctly and there is a workaround!

So this workaround only seems to work for the sample repo david.grant/reproduce-issue, doesn't work for our company's private repos. Rebase doesn't seem to work at all for all our repos.

@david.grant has the MR already been rebased, from looking at the source branch, or not?

I fixed it locally by adding 'GL_REPOSITORY'=> Gitlab::GlRepository.gl_repository(project, false) to https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/services/merge_requests/working_copy_base_service.rb#L52

It seems like we just forgot to set this variable for all the pushes. We covered web and ssh protocols but forgot that rebase feature has its own push!

PS The variable has been forgotten here https://gitlab.com/gitlab-org/gitlab-ee/commit/f4e6aba1bbeca043a29b4903cef2f5b99a1faac3 /cc @eReGeBe

Good catch @vsizov, sorry for missing that. Why did this not fail on the specs?

@eReGeBe I think this was pretty natural to forget:

• As @smcgivern noted, we have to refactor it.
• Specs do not cover it enough.

@eReGeBe this is pretty much my fault, and it even confused me (as you can see above!). We don't have specs that when we do a push from one service, the following PostReceive worker can act on that push - which is probably why we should try to avoid git push from services

mentioned in merge request !2531 (merged)

https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2531

closed via merge request !2531 (merged)

mentioned in commit 23e74a47

mentioned in commit 937d3ce6

Any estimate on when this gets released as a part of an official version? We're hitting this issue pretty hard, and it's the second time a GitLab update broke our workflow just since the beginning of July...

@rotemy I'm sorry about that - it will be in 9.4.2 https://gitlab.com/gitlab-org/gitlab-ce/issues/35589

Very grateful to see the fast fix on this one, thank you to everyone involved in that.

I don't think it's been mentioned yet: How did this get past release testing? Surely there must be at least one test (whether automatic or manual) that would pick up a major gitlab-EE feature like rebase not working at all? Is a test now in the process of being added?

mentioned in merge request release-tools!171 (merged)

@jogux you're right. The problem here is that, from the perspective of the service doing the rebasing, rebasing was working fine - the source branch was rebased. However the push that caused was not able to be processed by GitLab correctly, meaning the merge request wasn't updated with the new HEAD SHA.

This is quite fiddly to write an automated test for, and we would be better off removing the push completely (https://gitlab.com/gitlab-org/gitlab-ee/issues/1604) which would also be much easier to write a spec for.

For now, I've created an MR to add this to the QA checklist for the monthly release: https://gitlab.com/gitlab-org/release-tools/merge_requests/171

mentioned in issue #3019 (closed)

Sounds great, thank you once again Sean!

@smcgivern I think that is the right assessment of the problem in some cases but in our cases the rebase is failing. My hope is that once #1774 (closed) is fixed we'll be able to see what the problem is.

Just realized this bug is closed but would respectfully request it be re-opened as the original problem this issed was filed for I don't think is fixed.

My comment from above:

So this workaround only seems to work for the sample repo david.grant/reproduce-issue, doesn't work for our company's private repos. Rebase doesn't seem to work at all for all our repos.

@david.grant sorry about that - I suspect that is in fact related, but to check that I'll need some more information. Could you open a new confidential issue so that we can discuss there, please?

@smcgivern see #3039

@smcgivern Looks like it does work now! In one case today, the rebase was still actually failing (not just appearing to fail in the UI), I then closed and re-opened the MR, then re-ran the rebase and then it worked.

Thanks all for your work on this issue

@david.grant I'm glad it was fixed, sorry it was broken for so long

(For the record, I think this was broken on your MR because it was broken from a previous rebase, so new rebases also wouldn't work ... that kind of cascading failure is very painful, and we'll try to make things not break so hard in future.)

Just as a heads up, this still isn't working for us in 9.4.2. I think it's a completely different issue though, but another 9.4.x regression, I have no idea if it's related or not. The error is:

July 31, 2017 15:54 -> ERROR -> MergeRequests::RebaseService error (PROJECT/REPO!426): Failed to push rebased branch with `/opt/gitlab/embedded/bin/git push -f origin fix-missing-google-drive-files`:
July 31, 2017 15:54 -> ERROR -> MergeRequests::RebaseService error (PROJECT/REPO!426): remote: GitLab: You are not allowed to force push code to a protected branch on this project.
To /var/opt/gitlab/git-data/repositories/USER/REPO.git
! [remote rejected] fix-missing-google-drive-files -> fix-missing-google-drive-files (pre-receive hook declined)
error: failed to push some refs to '/var/opt/gitlab/git-data/repositories/USER/REPO.git'

The odd thing is the USER/REPO.git (ie. the 'from' repo) repository doesn't have any protected branches. It still fails if I unprotect all branches in the destination, but the error changes to:

July 31, 2017 16:05 -> ERROR -> MergeRequests::RebaseService error (PROJECT/REPO!426): remote: GitLab: Failed to authorize your Git request: internal API unreachable
```

I suspect the difference for us is we raise merge requests from forks (rather than from branches in the main project).

(raised with support under https://support.gitlab.com/hc/en-us/requests/80795?page=1 )

@jogux Can you create a separate issue? it's definitely a different one.

That would it be also good if you can create a public repo on GitLab.com where we can reproduce it.

@vsizov unfortunately I'm way too busy just now; the previous regressions in 9.4.x cost me a couple of days work and I desperately need to catch up. I hope support will do both those tasks fairly quickly.

(I use to open things as issues rather than going to support, but then my issues just got ignored... so I just go to support instead where we're at least meant to have an SLA. IMHO it'd be good if issues from paying customers got treated as if they were support tickets in terms of SLA etc... but I digress.)

@arihantar ^

@jogux I appreciate your inputs on the issue would also like to refer if you create an issue next time just let the support know so we can add appropriate tags, which would indicate to our Development team that it is an EE/paying customer asking for it.

@arihantar okay, thanks, I'll try that next time. I hope you can streamline that process sometime, seems daft I don't have permission to add the 'paying customer' tag myself, or even that it's not automatically added!

mentioned in issue #3071 (closed)

This is still not working for us in 9.4.2 @vsizov @smcgivern - anything we can do here?

@chauffer Are you sure you're not using forks like https://gitlab.com/gitlab-org/gitlab-ee/issues/3071 describes?

Only logs we can find are:

grep: /var/log/gitlab/: Is a directory
root@gitlab:/# grep -i rebaseservice /var/log/gitlab/ -R
/var/log/gitlab/gitlab-rails/githost.log:August 03, 2017 10:05 -> ERROR -> Merge
Requests::RebaseService error (group/repo!1289): Failed to rebase branch wi
th `/opt/gitlab/embedded/bin/git pull --rebase /var/opt/gitlab/git-data/reposito
ries/group/repo.git master`:
/var/log/gitlab/gitlab-rails/githost.log:August 03, 2017 10:05 -> ERROR -> Merge
Requests::RebaseService error (group/repo!1289): First, rewinding head to r
eplay your work on top of it...
/var/log/gitlab/gitlab-rails/githost.log:August 03, 2017 11:26 -> ERROR -> Merge
Requests::RebaseService error (group/repo!1289): Failed to rebase branch wi
th `/opt/gitlab/embedded/bin/git pull --rebase /var/opt/gitlab/git-data/reposito
ries/group/repo.git master`:
/var/log/gitlab/gitlab-rails/githost.log:August 03, 2017 11:26 -> ERROR -> Merge
Requests::RebaseService error (group/repo!1289): First, rewinding head to r
eplay your work on top of it...
root@gitlab:/#

@vsizov Yep, not forks.

Could we reopen this issue?

@chauffer this is a different issue. The problem previously was that the rebase was not updating the merge request. The problem in your case is that git pull --rebase is failing. Please open a new issue describing what happens when you try that locally - if there are conflicts, you will need to rebase locally rather than through the UI.

@smcgivern Rebasing locally works fine and I'm able to do it without any conflicts.

Regardless, shouldn't conflicts be handled in GL as well?

mentioned in issue #3085 (closed)

@chauffer no, we can only handle conflicts when merging. When rebasing, you can have conflicts when applying an arbitrary number of commits, which means that we'd have to stop the background worker each time and ask the user for input

Aha, I see, well; this doesn't seem to be the case here.