Skip to content

Update Unicorn WorkerKiller memory limit to 400M ~ 450M, 200 - 250M is not enough.

username-removed-198553 requested to merge huacnlee/gitlab-ce:patch-2 into master

https://staging.gitlab.com/gitlab-org/gitlab-ce/issues/42026

Too small a memory limit will restart Unicorn workers frequently, slowing down App response time.

I have upgraded GitLab 8.3 last week, the response time of users' feedback is slower than before.

And this few days, I have checked the GitLab source codes, production server environment, not finding anything...

The requests runtime always have 1s ~ 2s increased, after about 3~5 requests.

Today, I found some error messages in log/unicorn.stderr.log

W, [2016-01-14T16:35:48.335035 #21670]  WARN -- : #<Unicorn::HttpServer:0x007f0dcbfd3c18>: worker (pid: 21670) exceeds memory limit (281615360.0 bytes > 240519332 bytes)
W, [2016-01-14T16:35:48.335234 #21670]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 21670) alive: 29 sec (trial 1)
I, [2016-01-14T16:35:48.360816 #30515]  INFO -- : reaped #<Process::Status: pid 21778 exit 0> worker=18
I, [2016-01-14T16:35:48.378316 #24029]  INFO -- : worker=18 ready
I, [2016-01-14T16:35:48.610775 #30515]  INFO -- : reaped #<Process::Status: pid 21670 exit 0> worker=16
I, [2016-01-14T16:35:48.627959 #24035]  INFO -- : worker=16 ready
W, [2016-01-14T16:35:49.491209 #22202]  WARN -- : #<Unicorn::HttpServer:0x007f0dcbfd3c18>: worker (pid: 22202) exceeds memory limit (280159744.0 bytes > 258389605 bytes)
W, [2016-01-14T16:35:49.491382 #22202]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 22202) alive: 25 sec (trial 1)
I, [2016-01-14T16:35:49.776747 #30515]  INFO -- : reaped #<Process::Status: pid 22202 exit 0> worker=13
I, [2016-01-14T16:35:49.792827 #24097]  INFO -- : worker=13 ready
W, [2016-01-14T16:35:54.087233 #23046]  WARN -- : #<Unicorn::HttpServer:0x007f0dcbfd3c18>: worker (pid: 23046) exceeds memory limit (269072896.0 bytes > 253336298 bytes)
W, [2016-01-14T16:35:54.087394 #23046]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 23046) alive: 16 sec (trial 1)
W, [2016-01-14T16:35:54.163136 #22690]  WARN -- : #<Unicorn::HttpServer:0x007f0dcbfd3c18>: worker (pid: 22690) exceeds memory limit (275195392.0 bytes > 238146923 bytes)
W, [2016-01-14T16:35:54.163284 #22690]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 22690) alive: 22 sec (trial 1)
I, [2016-01-14T16:35:54.401420 #30515]  INFO -- : reaped #<Process::Status: pid 23046 exit 0> worker=10
I, [2016-01-14T16:35:54.436135 #30515]  INFO -- : reaped #<Process::Status: pid 22690 exit 0> worker=1
I, [2016-01-14T16:35:54.437998 #24337]  INFO -- : worker=10 ready
I, [2016-01-14T16:35:54.452401 #24340]  INFO -- : worker=1 ready
W, [2016-01-14T16:36:01.059089 #22534]  WARN -- : #<Unicorn::HttpServer:0x007f0dcbfd3c18>: worker (pid: 22534) exceeds memory limit (303286272.0 bytes > 248052412 bytes)
W, [2016-01-14T16:36:01.059290 #22534]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 22534) alive: 25 sec (trial 1)
I, [2016-01-14T16:36:01.319918 #30515]  INFO -- : reaped #<Process::Status: pid 22534 exit 0> worker=22
I, [2016-01-14T16:36:01.333799 #24781]  INFO -- : worker=22 ready
W, [2016-01-14T16:36:02.449341 #22315]  WARN -- : #<Unicorn::HttpServer:0x007f0dcbfd3c18>: worker (pid: 22315) exceeds memory limit (273725440.0 bytes > 248645964 bytes)
W, [2016-01-14T16:36:02.449501 #22315]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 22315) alive: 35 sec (trial 1)
I, [2016-01-14T16:36:02.745377 #30515]  INFO -- : reaped #<Process::Status: pid 22315 exit 0> worker=2

Unicorn Killer killed worker, and restart, so requests slowing down.

Same issue in gitlab.com, check online server log/unicorn.stderr.log

Edited by Victor Wu

Merge request reports