502 timeouts in creating merge requests

As reported by @markglenfletcher and @balameb in https://gitlab.com/gitlab-org/gitlab-ce/issues/13912#note_26319939, some users are experiencing 502 timeouts when creating merge requests in 9.0.

With the help of profile_url.rb, you can see that for the request @markglenfletcher linked, 92% of the time is spent just retrieving each patch:

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10251 should help, but loading the JSON data may still time out in this case.

Note that the repo is 1.4+ GB, which may explain why it takes so long to generate this data.

Nope, size of repo shouldn't count here, GL should handle it quicker. Optimization sucks here.

@stanhu how does this compare to the MR create page, and the compare page, for that pair of branches?

@smcgivern We see the same timeouts with the compare branch. I'll admit I was confused over the "From" and "To" nomenclature of the compare page (which is thankfully being resolved in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5972#note_24637085).

I think what @yorickpeterse said a year ago holds true more than ever:

The only way to optimize code like this is to either use a faster storage mechanism, or run it directly on the physical disk(s) storing the Git repository. The latter in turn requires some kind of API/service to run on said servers that takes a Git action to run (e.g. via HTTP) and spits out the result.

Rugged::Diff#each and Rugged::Diff#each_patch can be quite slow (up to 30 seconds).

I think this is case where Gitaly will come into play, unless we can somehow optimize libgit2 or NFS. In looking at the strace logs, I believe it just takes a long time to seek/mmap big files over a networked file system, just as we saw in https://gitlab.com/gitlab-org/gitlab-ee/issues/1811#note_26109097. If you look at the strace logs, you see large blocks of time taken like this:

04:17:53 mmap(0x7f87d1800000, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f87ca200000
04:17:53 munmap(0x7f87ca200000, 8388608) = 0
04:17:53 mmap(NULL, 20971520, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f87c9600000
04:17:54 mmap(0x7f87caa00000, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f87c8e00000
04:17:54 munmap(0x7f87c8e00000, 8388608) = 0
04:17:54 mmap(NULL, 29360128, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f87c7a00000
04:17:56 mmap(0x7f87ca200000, 25165824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f87c6200000
04:17:56 munmap(0x7f87c6200000, 25165824) = 0
04:17:56 mmap(NULL, 67108864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f87c3a00000
04:18:00 mmap(NULL, 83886080, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f87bea00000
04:18:02 madvise(0x7f87d0a0d000, 1052672, MADV_DONTNEED) = 0

libgit2 is trying to memory-map large pack files over NFS, which causes a lot of round-trips and network transfer. The more files there are in the diff, the more times we have to do this dance.

/cc: @andrewn

@stanhu thanks. I can just about tell from the screenshot that the problem appears to be in getting the changes count (?), as we load the changes tab async on the new MR page. I agree with everything in your previous comment, but is there anything here we think was caused by 9.0 specifically?

@smcgivern Probably not. Good call on the changes count. Perhaps we should be pushing that calculation when we update the merge request in Sidekiq?

@stanhu we don't do anything in Sidekiq for creating an MR at the moment, but maybe we could

Good point, we have to do this on-the-fly. It seems to me we really do need to rely on fast diff generation via Gitaly.

mentioned in issue #27381 (closed)

added Gitaly label

FYI @stanhu this has already been prioritised and some of the analysis work is already underway: see https://gitlab.com/gitlab-org/gitaly/issues/159.

We were blocked on https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1522 and https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10291 but these are cleared now so we'll recommence our work.

mentioned in issue gitaly#159 (closed)

Today I encountered this problem. Push branch -> New merge request -> loading.. and 502 -> Reload -> Merge request edit page -> Save -> Done.

mentioned in issue #36427 (closed)

marked this issue as related to #36427 (closed)

Adding description from https://gitlab.com/gitlab-org/gitlab-ce/issues/36427 below, as we've had this lots of times in the last week.

---

What

Error 502 when creating a merge request through the UI or API. Some of these are on the gitlab-ce project.

For the UI I've had 502s for #new, when changing source branch and might also have had it for create. For the API it would have been the equivalent of #create.

When

Many times in the last week

• @rdavila had this in https://gitlab.slack.com/archives/C101F3796/p1502726935746318

• @marin in https://gitlab.slack.com/archives/C101F3796/p1502715408307258

• @jamedjo in https://gitlab.slack.com/archives/C101F3796/p1502463184612669

• @jivanvl in https://gitlab.slack.com/archives/C101F3796/p1502461873850261

• @ClemMakesApps in https://gitlab.slack.com/archives/C101F3796/p1502100761221307

• @jameslopez in https://gitlab.slack.com/archives/C101F3796/p1502100768223429

• @yorickpeterse in https://gitlab.slack.com/archives/C02PF508L/p1502983983000601

• @jneen in https://gitlab.slack.com/archives/C02PF508L/p1502909638000107

This may or may not correlate with high load. A few times there have been questions about 502s for merge requests a couple of mins before a more general outage.

marked this issue as related to #32844 (closed)

I encountered again. Push branch -> New merge request -> Merge request edit page -> Save -> loading.. and 502 -> Reload -> loading.. and 502 -> Reload -> Done.

According to https://performance.gitlab.net/dashboard/db/rails-controllers?orgId=1&var-action=Projects::MergeRequests::CreationsController%23new&var-database=Production&from=now-7d&to=now as much as 12s is spent on Rugged::Diff#find_similar!. @ahmadsherif does gitaly_commit_raw_diffs cover this?

@eReGeBe It covers renames only, not copies.

@ahmadsherif @andrewn do we have then a migration for this case?

@eReGeBe we tend to refer to migrations by their GitLab::Git migration site. Do you know where Rugged::Diff#find_similar! is being called from in our codebase?

@andrewn Gitlab::Git::Repository#diff_patches. I don't see it on the migration board, so I'm guessing we haven't include it. It looks easy to do with gitaly-ruby.

I was able to create a very simple MR only after a fifth try. Should we assign someone to this issue @DouweM @smcgivern @andrewn?

@eReGeBe it needs to be migrated and with gitaly-ruby it is now possible. But the way the code is structured makes it hard as it is. Too many lazy interactions.

@grzesiek see https://gitlab.com/gitlab-org/gitlab-ce/issues/32844#note_37956016 and below. We could pick the MR resulting from that into 9.5, and maybe https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13416?

mentioned in issue release-tools#109

mentioned in issue gitaly#510

Gitlab::Git::Repository#diff_patches. I don't see it on the migration board, so I'm guessing we haven't include it. It looks easy to do with gitaly-ruby.

Created https://gitlab.com/gitlab-org/gitaly/issues/510 in the Gitaly tracker to track our migration

I ran a profile that seemed to take a long time to generate for the CE repository:

http://profiler.gitlap.com/20170830/7e1c6181-43d3-43bb-a894-9c63c4687727.html.gz

One thing that stuck out was that 19% spent on retrieving suggested approvers. I wonder if this is blocking the whole render pipeline: https://gitlab.com/gitlab-org/gitlab-ee/issues/2080

mentioned in issue #37477

marked this issue as related to #32352 (moved)

moved to gitlab#9003

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue