Timeout on branches page

This page seems to consistently time out for me. https://gitlab.com/python-devs/python/branches

Reassigned to @yorickpeterse

@yorickpeterse please look into this

/cc @dzaporozhets

/cc @jnijhof

Added ~18308 label

What I noticed so far from quickly looking into this:

On gitlab.com we're reporting a total of 35 branches, yet when I clone the repository I only end up having 3 branches. Loading the branches page locally works fine with those 3 branches.

The branches page basically works as following:

1. For every branch in the repository, get the latest commit
2. Check if this commit has been merged into the default branch
3. Continue until we've had all branches

From the top of my head I can see two main problems here:

1. When there are a lot of branches comparing the latest commit of these branches might be expensive. The sheer number of commits might further degrade performance.
2. The NFS server (or Azure) is not performing as well as it should.

@warsaw Is there another Git mirror of this repository that I can clone, hopefully with the 35 branches? That way I can test this locally to see if it's a problem with the application or our infrastructure.

Also, cloning this repository via HTTPS takes forever while cloning over SSH is blazing fast. This could be an issue with Workhorse horsing around too much, I doubt it's related to the NFS (or anything in between) as SSH would also suffer from it if this were the case.

Take a look here: https://github.com/python/cpython

They may have done a better job of the hg->git conversion than I did. The upstream official hg repository is here: https://hg.python.org/cpython/

I ended up cloning the GitHub mirror, the branches page loads in about 4-5 seconds locally. Looking at the output of our profiler I'm not seeing any database queries that might be problematic, suggesting this is indeed a problem with our Git interaction code or infrastructure.

The profiler suggests most of the time is spent in app/models/repository.rb line 496, which is the code used for comparing branches with the default branch (to see if they've been merged). The code is as following: https://gitlab.com/gitlab-org/gitlab-ce/blob/aec9f211e534900f602e769dcdd6f69730849f92/app/models/repository.rb#L491-500

While the root reference of the default branch is already cached, the output of the commit method is not. Memoizing this could at least reduce the amount of Git commands by N with N being the amount of branches available. I'll take a look at this on Monday & Tuesday.

Milestone changed to 8.3

I put 8.3 because its quite a serious problem if you cant load branches page

Sorry, haven't had time yet to further dig into this. I'm currently swamped by !1790 (merged) which requires many careful steps (and a lot of focus) to not break everything.

@warsaw Is there any form of deadline or time period when you folks will decide on what to use? I'm working on !1790 (merged) and I'd like to finish that first if possible, I hope to get this done the next 2 days. If the branches page is (more) urgent I can move things around, though the problem might be harder to fix if it's infrastructure related.

As a perhaps somewhat crazy alternative we could try setting up a githost.io account. I'm not sure what version we're currently running here, but it would allow testing without the gitlab.com infrastructure affecting things. @sytses would this be an option?

@yorickpeterse It is an option but I think they prefer GitLab.com, and I think that makes more sense too.

Roughly, by end of November we're going to evaluate the options and see what critical features or ui are missing. Then by mid-December we'll try to figure out how to work around those or a fix timeline. A final hosting decision will happen by January 1. Those are rough estimates and various things could push those back. My best guess is that the bulk of the evaluation will happen on or during the upcoming long USA Thanksgiving weekend.

@warsaw OK, then I'll try to look into this later this week, or at least before the 26th of November.

Our goal --if PEP 507 is accepted-- would be to have a hosted solution of GitLab at git.python.org, and ideally we wouldn't have to maintain the instance. We have a ton of infrastructure to maintain and so few resources that one of the main reasons to move code hosting is to reduce the load on our volunteer sysadmins. Right now we self-host hg and Reitveld for code and review (and Roundup for bugs, but we're keeping that). I am a proponent for moving to git and I think GitLab CE is a perfect fit for us based on features and the Open Source licensing.

But I don't get to decide so I'm just trying to help make this the obvious choice!

@warsaw Interesting enough it seems the branches page no longer times out, at least I just managed to load it within a reasonable time frame.

Indeed! Thanks for the great work. I'll let you guys close the issue.

@warsaw Since we didn't fix anything specifically for this I'll leave this issue open for a few more days in case the problem resurfaces.

Milestone changed to 8.4

I spent the morning looking into this to see if I could come up with a definitive solution. Sadly I can't find a solution on the application level. The code basically does the following:

1. Iterate over all branches in a project
2. Check if the latest commit of a branch is in the root branch (usually "master")
3. If so, set the "merged" status to true

Retrieving the latest commit from the root branch is already cached, thus there's little to do here. Iterating over the branches can't be done in any other way as Rugged/libgit2 doesn't provide a way of comparing multiple branches in a single call. As a result the above setup is currently the only possible solution.

One possible but rather complex solution is to set the "merged" status of a branch whenever a commit is pushed to it. Whenever a commit is force pushed to the root branch the "merged" status of all branches would also be rebuilt (e.g. a git push -f could remove a commit, thus leading to a branch no longer being merged). This setup however would require extra Sidekiq code, could lead to race conditions (push, Sidekiq starts working, another push, now the "merged" state might no longer be accurate). It would also require storing the branch names in PostgreSQL/MySQL as Git doesn't allow attaching of arbitrary meta-data to branches as far as I'm aware.

Caching the views wouldn't help either. If the first request leads to a timeout we'd never get to actually build the cache, thus we'd keep getting timeouts. Caching would also mean having to invalidate it upon every push, meaning it won't really help for busy projects.

The only viable solution I can think of is to simply make interacting with the Git repositories as fast as possible. Since this is more of an infrastructure related problem (certainly not something we can fix in GitLab CE itself) I'll be closing this issue.

Status changed to closed

mentioned in issue #12619 (moved)

Mentioned in merge request !5751