Merge request stuck on `locked` state although it got merged

Summary

There's a known case where a MR got merged, but it kept stuck on locked state.

Merge request https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11620

Merge commit on master https://gitlab.com/gitlab-org/gitlab-ce/commit/4c8d478eefc4b50ffc26f0d1ce159470ae9f0105

Steps to reproduce

The only information at the moment is that the user clicked on Merge (cc @kushalpandya)

(That makes me wonder if we're actually running on a race-condition there)

What is the current bug behavior?

• Merge request persists locked state and does not switch to merged state
• Merge request FE presents Locked state

What is the expected correct behavior?

• Merge request should correctly switch from locked to merged state
• Merge request FE should present Merged state

This bug happened on GitLab.com

cc @smcgivern @DouweM

changed the description

@oswaldo Where in the code is the change from locked to status supposed to happen? Why would that not happen? :)

@DouweM

We wrap the after_merge (where it will switch from locked to merged) on MergeRequest#in_lock_state call here:

https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/services/merge_requests/merge_service.rb#L26-31

And, even though it fails on yielding the block, we ensure on MergeRequest#in_locked_state that it'll at least go from locked to open.

That said, we seem covered. But still couldn't figure out this edge case. Any thoughts?

@oswaldo Note that ensure may not be run if Sidekiq hard-kills the worker that executes this merge because for some reason.

commit didn't time out or anything since the merge went through, but since the status didn't change to merged at the source branch wasn't deleted, maybe after_merge was never run? Does that seem plausible? I'm not sure why Sidekiq would kill the worker though.

/cc @stanhu

I thought we stored the Sidekiq job ID, but it looks like we only do that for the import tasks. Perhaps we can find it using Kibana.

The MR ID is 3568445, but I see no MergeWorker in Kibana associated with it.

Issue for storing the job ID: https://gitlab.com/gitlab-org/gitlab-ce/issues/31207

Although we don't have logs (seemingly) for this case, and it's slightly different, https://gitlab.com/gitlab-org/gitlab-ce/issues/31207#note_29047861 might be useful. Unfortunately it's not as Douwe suggested, because the Sidekiq worker was not killed, but continued to process jobs.

I don't really have anything useful here, but https://gitlab.com/gitlab-org/gitlab-ce/issues/31207#note_30732833 sounds like a good suggestion, even if it's not the cause of this issue.

Our self hosted Gitlab CE instance is also affected by this issue since our update from 9.1.4 to 9.2.2 this morning.

Since the upgrade a lot (but not all) MR are stuck in the locked state although the merge is successfully achieved on the target branch

To add some info that I hope may be useful for a fix we've noticed that MR are not unlocked because the queue jobs are stuck in a "running" state indefinitely in sidekiq. Worse : when the size of the queue reach 25 no more async tasks are processed.

The only way we found to unlock this state is to manually stop the sidekiq deamon with this command : gitlab-ctl stop && gitlab-ctl sidekiq start

or by forcing the a restart of the whole gitlab instance by running :

sudo service gitlab reload

But this trick leave the unprocessed MR in a locked state for ever unless we run manually this quite dirty hack : https://gitlab.com/gitlab-org/gitlab-ce/issues/18048#note_30001119

Any help will be very welcome guys

Best regards

We have the same problem. A merge request is stuck in Locked state although it was successfully merged. I cannot fix this myself since we use GitLab SaaS.

Closing in favour of https://gitlab.com/gitlab-org/gitlab-ce/issues/31207, as this isn't a new issue.

We currently unlock after 24 hours, but I hope we'll get to https://gitlab.com/gitlab-org/gitlab-ce/issues/31207 by 9.5.

closed

I have the same problem. It's the first merge request in the project