can't do any merge requests

Hi,

i already posted https://gitlab.com/gitlab-org/gitlab-ce/issues/831 but new verions have same problem.

sometimes sidekiq (yes, i believe it is) freezes.

now i increased number of workers in 4 times (100 workers). and most every day i see this: sidekiq 3.3.0 gitlab [100 of 100 busy].

where is the problem?

gitlab has 7.8.3 version.

For the users this problem affected the next:

• Nobody could add a new Merge Requests. The GitLab doesn't show a new created branches to do this.
• When the branch is added/removed then the list of branches is not updated on appropriate screen.
• Merge Submit is failed. If I try to do this just to see the infinity progress.
• Email notifications does not worked. So, I could not to know about new comments OR merge requests (for the old branched).
• etc. problems.

I've seen similar problem when someone pushed branch with unfinished rebase.

@sashk have you solved it?

Yes, I finished rebase. Did you try that?

@sashk finished it where? I'm a person who maintain gitlab in our company. And I can't catch this bug on mine repos.

i know that "freezing" happens on different repositories and maybe different reasons. and i can't say for all developers.

but i want to know how to solve this bug once and forever)

Here is problem I had:

1. Developer branched of branch master and started his work.
2. Code review happened, developer was asked to rebase and smash couple commits.
3. Developer started doing rebasing, but got stopped by conflict.
4. Developer resolved conflict, but forgot to do git rebase --continue and pushed to gitlab.

What I've got after that:

1. I couldn't locate branch in branch list.
2. I couldn't make merge request from that branch.
3. when cloning I was getting weird results.

How I fixed?

I've asked developer to do git rebase origin master and made sure that he successfully finished his rebase before git push -f ....

Hope that helps.

But I think, GitLab should somehow report if it got branch in weird state.

@sashk ok. Thank you, i'll try to check your solution

@sashk I wasn't able to reproduce the problem by following your steps. What I get is that git doesn't push the rebase in progress at all. Only the pre-rebase state is pushed. So there is no way to mess up the push unless you specifically request git to do so, i.e. by creating a new commit during the rebase and then issueing the git push --force origin HEAD:branch command, which is unlikely to happen by mistake.

@dmedvinsky It's exactly what my users did.

some updates:

new versions have same "bug" - sometimes sidekiq freezes and collects queues... i tried to extend up to 300 workers... doesn't help. but, we thought - if some jobs blocks each other, so lets try to run only one worker.

so 3 days passed without any problems.

Status changed to closed

Status changed to reopened

problem stil persist.

have some screenshots:

• http://imageshack.com/a/img540/9684/9c6ynP.png
• http://imageshack.com/a/img912/1308/x4PRyG.png
• http://imageshack.com/a/img909/5428/dG0fIu.png
• http://imageshack.com/a/img910/3073/HRRCHD.png

yesterday updated to new 7.12.1 and promblem appeared today.

@mrPsycho did you solve your problem? im getting the same issue in 8.2.1.

@fernandoneto seems so.

we start sidekiq with -c 1, so only one worker starts and wrote script:

#!/usr/bin/python

from time import sleep
import subprocess
import os

import redis

pid = str(os.getpid())
pidfile = "/var/run/sidekiq-monitor.pid"

file(pidfile, 'w').write(pid)

command = ['/etc/init.d/gitlab', 'reload']
timeout = 120  # seconds
busyThreshold = 1
enqueuedThreshold = 100

r = redis.Redis(unix_socket_path='/var/run/redis/redis.sock')

while True:
    keys = r.keys('resque:gitlab:devcenter:*')
    if keys:
        for key in keys:
            busy = r.hget(key, 'busy')
            if busy and int(busy) >= busyThreshold:
                subprocess.call(command)
                break
    queues = r.smembers('resque:gitlab:queues')
    if queues:
        enqueued = 0
        for queue in queues:
            enqueued += int(r.llen("resque:gitlab:queue:{0}".format(queue)))
        if enqueued >= enqueuedThreshold:
            subprocess.call(command)
    sleep(timeout)

os.unlink(pidfile)

which works in background.

6 months without problems.

Status changed to closed

@mrPsycho In the new version of Gitlab v8.3.0 the sidekiq is updated in Gemfile from 3.x.x to 4.0.1. I will make some tests with this version and give you some feedback.

mentioned in merge request !14785