Revise Unicorn memory limits

We have some ongoing experiments with Unicorn memory limits on gitlab.com (please let me know if this is a duplicate issue). I am creating this issue as a reminder that we need to:

• update Omnibus default memory limits and its worker count formula in attributes/default.rb
• update gitlab-ce config.ru memory limit defaults
• update the RAM requirements in https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/requirements.md#memory

This issue replaces https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/1883 (a private 1-year old stalled MR).

cc @pcarranza @marin

Moved from operations#365 (closed)

Added ~366505 documentation critical labels

Thanks @jacobvosmaer-gitlab you are totally right.

@marin do you accept MRs for this?

@pcarranza Yes!

https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/requirements.md#memory should link to:

1. Omnibus default memory limits and its worker count formula in attributes/default.rb
2. update gitlab-ce config.ru memory limit defaults
3. what metrics we watch too see if it is too low

All of these limits should be equal to the limit that we use on GitLab.com. We should try if it can be lower than 600MB without affecting anything.

@marin let's start with this release setting the mix-max to 419430400 - 734003200, does that make sense?

The default limits are in MR already: https://dev.gitlab.org/gitlab/omnibus-gitlab/merge_requests/26

Merged, picking into stable

@JobV maybe we should say something about this?

mentioned in commit gitlab-org/omnibus-gitlab@8f688097

mentioned in commit gitlab-org/omnibus-gitlab@afc15c96

mentioned in commit gitlab-org/omnibus-gitlab@a375e61e

@jacobvosmaer-gitlab @pcarranza Is there any concern about this affecting GitLab on RasPi?

@dblessing The packages for RasPi are currently not being released because of gitlab-org/omnibus-gitlab#1337 (closed)

I was going to create a new issue, but this one already exists with good discussion. It's apparent that 2GB RAM with no swap doesn't work well with GitLab at the moment: lots of out of memory errors occur if you try to do anything (e.g. upgrade, backup, etc.). Sidekiq is capped at 1 GB, and 3 Unicorn workers consume about a 1 GB RAM without much activity. In addition to the items @jacobvosmaer-gitlab mentioned above, we need to:

• Raise GitLab memory requirements from 2 GB + no swap -> 2 GB + some swap (or 4 GB)

I think a good goal for the end of the year is to bring the GitLab memory requirements back down under the 2 GB RAM mark.

@pcarranza Can we have a performance specialist handle the memory requirements overhaul for 8.12?

Let's add it to the agenda @stanhu

Reassigned to @ahmadsherif

@ahmadsherif I'm assigning this to you as you are handling memory usage. Can you please update the documentation for the time being so it makes sense with what is actually needed now?

@pcarranza OK.

@pcarranza I tried applying some workload on a GitLab installation in a 2 GB droplet with no swap; 3 ruby scripts doing small pushes in parallel with 1 second pause between every push, along with brief browsing of the site itself. The droplet free memory went down kinda fast, to the point it caused 500 errors (due to workhorse not being able to run new processes) and/or 502 errors (because unicorn was killed as a whole). Adding a 1 GB swap helped eliminating the errors. Also tried higher workload (10 scripts instead of 3) on a 4 GB droplet with no swap, no 50x errors and the site feels OK, memory was decreasing but this time memory was reclaimed by (automatically) killing Sidekiq, unicron memory is kinda stagnant.

Should we update the docs to a minimum of 4 GB addressable memory?

This sounds fair @ahmadsherif thanks for working on it.

We could also say that we can make it work on 2Gb with 1Gb of swap, but that you can expect degradation to happen (not so much on SSD drives)

cc/ @stanhu

Yes, I think 4GB makes sense.

Mentioned in issue githost#24 (closed)

Assignee removed

Memory limits appear to have been merge long time ago by @ahmadsherif

I'm closing this, open a new one if we want to review again.

closed