502 while submitting a large merge request

With GitLab 8.4 (stable), I'm hitting an issue while submitting a merge request with about 400 files touched. The browser waits, then shows a 502 Proxy Error. I've watched the Unicorn worker using 100% CPU for about 30 minutes. When it stopped (it did), the merge request was not visible in GitLab, as it probably failed.

I've raised the ProxyTimeout option in Apache and the timeout option in GitLab. This doesn't seem to be a memory issue (less than 25% RAM, 0% swap).

The merge request doesn't contain anything really big, but there are a few minified JavaScript / CSS files, one of them is about 10 MB.

It created a 500 GB (yes, GB!) production.log (which, for obvious reasons, I will not include here), in which I can read Mysql2::Error: MySQL server has gone away. Because of its size, it's hard to get anything from it.

Additional information:

• "Compare branch and continue" is almost instantaneous.
• the repository is about 74 MB
• the content of production.log seems to be mostly source code from those minified JavaScript files, repeating themselves, like, forever
• most lines in the log file starts with: Mysql2::Error: MySQL server has gone away: UPDATE merge_request_diffs SET state = 'collected' (then the whole diff)

I've got the feeling that GitLab is trying to compare these one-line 10-MB files instead of considering them binary.

Added ~18308 label

/cc @yorickpeterse @joshfng

Specs GitLab is running on:

• 4-core Intel Xeon @ 3.10GHz
• 8 GB RAM
• 1 GB swap
• HDD 1 TB (~100 MB/s write)

GitLab is not the only process running on the server, but I made sure nothing was active during my tests. RAM usage never went over 25% and swap was totally unused.

I triggered the same merge request, but without the minified source files, and it did not take more than a few seconds.

Thinking out loud:

I believe @jacobvosmaer is working on some changes regarding the limit on the length of diffs, though I believe that's based on the number of lines being displayed. For minified source files (where everything is usually on a single line) we could maybe also provide a column based limit.

SmartGit (a popular Git client I use) has a column based limit of 10000 characters, which is more than enough to conclude that the file is not intended to be human-readable, and it considers the file as binary beyond that limit.

Current MR diff limits (which are implemented not entirely effectively) are indeed based on the number of files changed and the number of lines with changes. We could add a third limit, on the number of bytes in the text representation of the diff. (That is easier to implement than scanning a string to find the longest line.)

In https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2705 I am working on making the existing diff limits more effective. That may remove the need for a 'byte limit'. On the other hand it is perfectly sane to have it.

(What is ineffective about how diffs are limited at the moment is that if a diff is too large to be rendered it is still loaded into memory first in its entirety by GitLab, and just loading the diff into memory can already give a timeout.)

Hi,

Similar error here trying to merge minified CSS

Gitlab 8.5 - same problem when merge min CSS...

@akas84 @bbaprowski Is it possible for either of you to provide a file that triggers the problem? If not, can you tell us:

• How many lines there are in the CSS file
• What the largest amount of characters is on a single line
• The total file size

@yorickpeterse The problem is with one file:

~380 000 chars in one line, ~500KB

@bbaprowski Aha, so in that case our "diff limiter" probably doesn't help because it's line based, not line + column based.

Added diff label

Milestone changed to 8.6

@yorickpeterse Ok, but what can we do with that bug? Downgrade Gitlab to 8.3?

@bbaprowski I don't think there's anything you can do at this point since the problem has been around forever (as far as I can tell).

Added repository label

mentioned in merge request gitlab_git!72 (merged)

mentioned in merge request !3185 (merged)

Status changed to closed by commit 726f1efd

mentioned in commit 726f1efd

mentioned in commit f14ee2f2

Mentioned in commit pfjason/gitlab-ce@f14ee2f2

Mentioned in commit pfjason/gitlab-ce@726f1efd

Mentioned in issue #24566 (moved)

mentioned in commit test-nick/gitlab-ce@3bd25b79

mentioned in issue gitlab#7048