GitLab blows out production.log (fills up disk) if someone attempts to create a too-large merge request

Dup report of gitlab-org/gitlab-ce#2889 which seems to have been ignored.

It appears to be spinning continuously reporting the same mysql error in the log if user tries to create a merge request with diffs blob that is too large relative innodb log file size or if there is any other similar error on mysql refusing the query.

Mysql failure can blow out production.log - 35GB overnight in my case Underlying cause was an absolutely massive merge request (280+ MB insert into merge_request_diffs) that I'm sure exceeded max_allowed_packet setting. The symptom-causing problem in this case though is that gitlab spewed it repeatedly to production.log until the disk filled. Can you add a setting (preferably with a sane default) to truncate log messages of the form: Mysql2::Error: MySQL server has gone away: UPDATE merge_request_diffs SET st_commits = or similar - i.e. just cap them to a sane size unless user has specifically configured to log the entire event. This turned an individual specific failure (user wouldn't be able to submit that merge request) into a whole system failure due to the repeated log entries filling disk. Probably should also consider an option to set a size limit for merge request diffs that could be validated automatically by requesting the max size from the db driver.

Just got this again last night.... this behavior is really not ideal. In last nights case, it had to do with blob size limits on mysql due to a large patch. I'm not objecting to the error - it SHOULD error out - the problem is that gitlab is spinning on the failure, generating MASSIVE production.log file, in turn filling up the disk. Last night it wrote 83 GB of logs from a SINGLE overly-large merge request.

Simple change request for this - please don't dump ENTIRE contents of the blob in the mysql error into the log when it's >8K or something like that.

That will help, better would be to not busy-spin retry the same request - add a retry delay so it doesn't just sit there and spin retrying which is guaranteed to fill up any available space.

We experienced the same issue with postgresql! The log file fills up the entire disk space and stalls the server... Is there a solution yet?

We have same issue last night and gitlab service was unavailable. we use 8.2.2. Does anyone know why 'MYSQL server has gone away' log repeat until disk is full.

I think we should keep this discussion in one of the two issues created for this problem. I suggest we use #2889 (closed) as that one was created first and do no longer use this one (can someone close this issue as a duplicate? I don't seen an option for that).

Status changed to closed

Closing as duplicate of #2889 (closed)

mentioned in merge request !14785