Mysql failure can blow out production.log - 35GB overnight in my case
Underlying cause was an absolutely massive merge request (280+ MB insert into merge_request_diffs) that I'm sure exceeded max_allowed_packet setting.
The symptom-causing problem in this case though is that gitlab spewed it repeatedly to production.log until the disk filled.
Can you add a setting (preferably with a sane default) to truncate log messages of the form:
Mysql2::Error: MySQL server has gone away: UPDATE merge_request_diffs
SET st_commits
=
or similar - i.e. just cap them to a sane size unless user has specifically configured to log the entire event. This turned an individual specific failure (user wouldn't be able to submit that merge request) into a whole system failure due to the repeated log entries filling disk.
Probably should also consider an option to set a size limit for merge request diffs that could be validated automatically by requesting the max size from the db driver.