locking test

TODO (what I can currently think of):

add Puma support files
update init scripts for Puma
test Puma locally
enable Puma worker killer
test the Puma worker killer Gem to make sure it actually terminates workers
make sure the application's memory usage is stable enough (or we at least know where it's problematic)
update Omnibus & friends to support any new configuration files, PIDs, commands, etc

cc-ing @sytses @dzaporozhets @DouweM @rspeicher since you all might be interested in keeping an eye on this

cc-ing @jacobvosmaer and @jnijhof since you'll probably have to deal with any Omnibus changes

@yorickpeterse You might be interested in https://github.com/gitlabhq/gitlabhq/pull/9832 which patches GitLab for Puma

@razer6 Thanks, I'll cross-link things, it seems that MR also does more than just adding Puma support (e.g. https://github.com/gitlabhq/gitlabhq/pull/9832/files#diff-a8874066192f517e5362d2fc0c3db2c1R48).

Unrelated, resources that are useful for this:

Added 1 commit:

dd1ebe18 - Enable puma_worker_killer

Added 4 commits:

11763eec - Added Puma to the Gemfile
fda91966 - Added Puma example configuration file
5bba7a95 - Increase database pool sizes to 72 for production
56383043 - Enable puma_worker_killer

Added 1 commit:

84040fba - Removed Thin from the Gemfile

I don't see what is wrong with Unicorn. I am not sure if this is worth the effort @yorickpeterse .

@jacobvosmaer Unicorn is a waste of memory, see #3592 (moved) for more information/rationale/etc.

Added ~18308 label

@yorickpeterse Don't forget you need to use CXX/CC flags for building C API Ruby Extension, to improve the situation in production GCC/LLVM stack-smashing protection.

CentOS/RHEL used default this flags for building C/C++ application/packages-yum:

-O2 -march=native -fstack-protector-strong --param=ssp-buffer-size=4

-fstack-protector instead of -fstack-protector-strong to GCC 4.9 compilers below

-Wl,-z,relro,-z,now - [Only support for linux hardening runtime]

Refs:

Buffer overflow protection – GNU Compiler Collection (GCC) [Only support for linux hardening runtime]: https://wiki.debian.org/Hardening
Improve your software security with GCC Stack Protector Strong
RELRO: RELocation Read-Only
Code-Pointer Integrity – Fast and precise control-flow hijack protection
CloudABI

@denji0k Pardon my potential stupidity, but how is this related to getting GitLab to run in a multi-threaded fashion?

@yorickpeterse This applies to C-Ruby extensions (including puma) to reduce the number of leaks. Globally gem installed via gem(root), symlink will be created for puma in vendor/bundle.

If compiled without these flags, the possible leaks in all C-Ruby extensions to bypass ruby Garbage Collector.

I'm talking about C-Ruby things that will reduce the effect of exploitation of vulnerabilities and a significant decrease of leakage of the memory allocation unused.

First steps I'll be taking now that Puma is configured: memory testing. This process will basically be as following:

Grab 10 important pages (home page, project index page, project issues page, etc)
Send a certain number of requests to each page, measuring the total memory usage before and after all requests. Do this for both Unicorn and Puma.
Graph the results so it's easier to see any differences

The goal of this would be to see if there's any difference in memory growth between Unicorn and Puma. If Puma is (or appears to be) stable with this set I'll start testing a bunch of other pages as well to see how those behave.

@yorickpeterse I think it is a bad idea to completely remove Unicorn before you know Puma is stable and actually an improvement. The best way to put Puma to the test is to deploy it on gitlab.com.

We should not inflict the pain of switching web servers on all GitLab installations before we know it is worth it. cc @sytses

mentioned in issue #3592 (moved)

I think it is a bad idea to completely remove Unicorn before you know Puma is stable and actually an improvement. The best way to put Puma to the test is to deploy it on gitlab.com.

Nothing is being replaced for all users until we know for certain that what we're moving to is worth it. I can re-add Unicorn if that makes people happy, but with all the Azure shit going on I'm reluctant to do testing on gitlab.com until I know for certain Puma is stable (dev.gitlab.org is a different story).

@denji0k Sorry, I should've mentioned that I meant pages of GitLab CE/EE, not the forum and the likes (those all run different applications). Vegeta looks interesting, I'll keep it around, normally I just use siege (https://www.joedog.org/siege-home/).

Added 2 commits:

bb45ccaf - Enable puma_worker_killer
5f8a5a17 - Removed Thin from the Gemfile

I think it is important to ship this in a way that we use Unicorn by default. For example, use an environment variable in config.ru.

The reason to keep Unicorn by default is that we won't know if Puma is a good idea for GitLab until it is on gitlab.com.

After having had a call with @jacobvosmaer and @marin we'll go with the following:

We'll add support for Puma source code wise, hidden behind an environment variable (thus disabling it by default). Unicorn is still the default and officially supported way to go.
This MR will solely focus on adding Puma and testing if it's actually stable (including memory growth), anything related to thread-safety will be taken care of in separate merge requests as the scope otherwise becomes too big.
Should Puma be stable enough locally the above would allow us to test it on gitlab.com for a while (e.g. 2-3 weeks). We can then evaluate the results and decide if we want to officially switch.

I've updated the above list of TODO items accordingly, I'll also rename this MR and its description to match this.

Removed ~18308 label

Milestone changed to 8.3

Title changed from WIP: Multi threading application servers to WIP: Add experimental Puma support

Added 1 commit:

7c66a04e - Hide Puma behind an environment variable

I propose the following testing steps:

integrate a working Puma config in GitLab CE, with an environment variable (e.g. GITLAB_USE_PUMA) to switch between Unicorn and Puma
merge to master and turn on Puma on dev.gitlab.org -> see how it behaves in a lightweight production environment
in omnibus-gitlab, add an off-by-default 'puma' service that can be turned on instead of unicorn
experiment on gitlab.com by turning on Puma on one worker server only

If after all those steps Puma looks as stable as Unicorn is now, and if it is an improvement in some way, then we could replace Unicorn with Puma. But no sooner.

Added 1 commit:

234cd143 - Add experimental support for Puma

Added 1 commit:

ecdd6e17 - Add experimental support for Puma

Thanks @yorickpeterse I think we wrote the same thing at the same time.

To benchmark memory usage I created the repository https://gitlab.com/yorickpeterse/gitlab-memory-benchmarks which contains a bunch of shell scripts to take care of this.

To benchmark things I first disabled all the worker killing code so this can't mess things up. You can use the following patch for this:

diff --git a/config.ru b/config.ru
index a2525c8..d90b2a4 100644
--- a/config.ru
+++ b/config.ru
@@ -1,5 +1,6 @@
 # This file is used by Rack-based servers to start the application.

+=begin
 if defined?(Unicorn)
   require 'unicorn'

@@ -11,6 +12,7 @@ if defined?(Unicorn)
     use Unicorn::WorkerKiller::Oom, (200 * (1 << 20)), (250 * (1 << 20))
   end
 end
+=end

 require ::File.expand_path('../config/environment',  __FILE__)

diff --git a/config/environments/production.rb b/config/environments/production.rb
index 3316ece..c6c956e 100644
--- a/config/environments/production.rb
+++ b/config/environments/production.rb
@@ -32,7 +32,7 @@ Gitlab::Application.configure do
   # config.force_ssl = true

   # See everything in the log (default is :info)
-  # config.log_level = :debug
+  config.log_level = :error

   # Suppress 'Rendered template ...' messages in the log
   # source: http://stackoverflow.com/a/16369363
diff --git a/config/initializers/puma.rb b/config/initializers/puma.rb
index f737f07..aec3946 100644
--- a/config/initializers/puma.rb
+++ b/config/initializers/puma.rb
@@ -2,6 +2,7 @@ if ENV['USE_PUMA']
   require 'puma'
   require 'puma_worker_killer'

+=begin
   if Rails.env.production? or Rails.env.staging?
     PumaWorkerKiller.config do |config|
       config.ram       = 250
@@ -19,4 +20,5 @@ if ENV['USE_PUMA']

     PumaWorkerKiller.start
   end
+=end
 end

Next, start GitLab CE/EE in production mode using one of the following two commands (depending on the server you want to benchmark):

Unicorn:

RACK_ENV=production RAILS_ENV=production bundle exec unicorn config.ru -l 3000

Puma:

RACK_ENV=production USE_PUMA=1 RAILS_ENV=production bundle exec puma config.ru -b tcp://0.0.0.0:3000

Once booted up you can start the benchmarking process by running ./bench.sh in the benchmark repository. If you have gnuplot installed you can then plot the memory usage over time by running either plot_puma.sh or plot_unicorn.sh. You'll also need to have Qt5 installed, though this should come as a dependency of gnuplot.

On my laptop I get the following graphs:

Unicorn:

Puma:

Here we can see that while Puma baseline memory usage is slightly higher it does not appear to grow differently compared to Unicorn. Note that I spent exactly no time tuning either Unicorn and/or Puma, other than setting RAILS_ENV and RACK_ENV to "production" both run using their default configuration settings.

In terms of requests handling performance, the numbers are below. First a 5 second warmup is performed, then the application is benchmarked using 1 concurrent connection. Finally the application is benchmarked using 4 concurrent connections.

Unicorn

Warmup:

$ siege -b -t 5s -c 1 http://localhost:3000/gitlab-org/gitlab-ce
** SIEGE 3.1.1
** Preparing 1 concurrent users for battle.
The server is now under siege...
Lifting the server siege..      done.

Transactions:                     40 hits
Availability:                 100.00 %
Elapsed time:                   4.86 secs
Data transferred:               0.84 MB
Response time:                  0.12 secs
Transaction rate:               8.23 trans/sec
Throughput:                     0.17 MB/sec
Concurrency:                    0.98
Successful transactions:          40
Failed transactions:               0
Longest transaction:            0.70
Shortest transaction:           0.09

Benchmarking using 1 concurrent connection:

$ siege -b -t 30s -c 1 http://localhost:3000/gitlab-org/gitlab-ce
** SIEGE 3.1.1
** Preparing 1 concurrent users for battle.
The server is now under siege...
Lifting the server siege..      done.

Transactions:                    233 hits
Availability:                 100.00 %
Elapsed time:                  29.03 secs
Data transferred:               4.88 MB
Response time:                  0.12 secs
Transaction rate:               8.03 trans/sec
Throughput:                     0.17 MB/sec
Concurrency:                    1.00
Successful transactions:         233
Failed transactions:               0
Longest transaction:            0.30
Shortest transaction:           0.09

4 concurrent connections:

$ siege -b -t 30s -c 4 http://localhost:3000/gitlab-org/gitlab-ce
** SIEGE 3.1.1
** Preparing 4 concurrent users for battle.
The server is now under siege...
Lifting the server siege..      done.

Transactions:                    231 hits
Availability:                 100.00 %
Elapsed time:                  29.63 secs
Data transferred:               4.84 MB
Response time:                  0.51 secs
Transaction rate:               7.80 trans/sec
Throughput:                     0.16 MB/sec
Concurrency:                    3.98
Successful transactions:         231
Failed transactions:               0
Longest transaction:            0.84
Shortest transaction:           0.12

Puma

Warmup:

$ siege -b -t 5s -c 1 http://localhost:3000/gitlab-org/gitlab-ce
** SIEGE 3.1.1
** Preparing 1 concurrent users for battle.
The server is now under siege...
Lifting the server siege..      done.

Transactions:                     38 hits
Availability:                 100.00 %
Elapsed time:                   4.63 secs
Data transferred:               0.80 MB
Response time:                  0.12 secs
Transaction rate:               8.21 trans/sec
Throughput:                     0.17 MB/sec
Concurrency:                    0.98
Successful transactions:          38
Failed transactions:               0
Longest transaction:            0.66
Shortest transaction:           0.09

Benchmarking using 1 concurrent connection:

$ siege -b -t 30s -c 1 http://localhost:3000/gitlab-org/gitlab-ce
** SIEGE 3.1.1
** Preparing 1 concurrent users for battle.
The server is now under siege...
Lifting the server siege..      done.

Transactions:                    264 hits
Availability:                 100.00 %
Elapsed time:                  29.33 secs
Data transferred:               5.53 MB
Response time:                  0.11 secs
Transaction rate:               9.00 trans/sec
Throughput:                     0.19 MB/sec
Concurrency:                    1.00
Successful transactions:         264
Failed transactions:               0
Longest transaction:            0.19
Shortest transaction:           0.10

4 concurrent connections:

$ siege -b -t 30s -c 4 http://localhost:3000/gitlab-org/gitlab-ce
** SIEGE 3.1.1
** Preparing 4 concurrent users for battle.
The server is now under siege...
Lifting the server siege..      done.

Transactions:                    273 hits
Availability:                 100.00 %
Elapsed time:                  29.68 secs
Data transferred:               5.72 MB
Response time:                  0.43 secs
Transaction rate:               9.20 trans/sec
Throughput:                     0.19 MB/sec
Concurrency:                    3.97
Successful transactions:         273
Failed transactions:               0
Longest transaction:            0.55
Shortest transaction:           0.11

Few more additions after talking this over:

Please keep any discussions about the benefits/drawbacks of Puma, multi-threading, etc in the corresponding issue (#3592 (moved)). This way we can keep the comments in this MR focused on the actual changes and their impact.
This will be "parked" for the time being until we have better insight in where we actually leak memory, how much, etc. This will most likely require taking care of issues such as #2936 (closed) first. I'll also be creating a separate meta issue for this.
GitHub PR https://github.com/gitlabhq/gitlabhq/pull/9832 adds a similar set of changes, once the time comes I think it would be nice to cherry-pick those (and give the author credit where credit is due) instead of hijacking it.

Added blocked label

Milestone removed

mentioned in issue #3700 (closed)

@denji0k Please see my comments above, both Puma and Unicorn were started in production mode. I'm not sure what you're meaning with the Gemfile and diff, no Gems were downgraded in this MR.

@yorickpeterse I'm closing this to reflect it is blocked.

Status changed to closed

mentioned in issue #26396 (closed)

@yorickpeterse maybe the time for Puma support has come, I'm reopening this. /cc @sitschner

reopened

mentioned in issue #38242

In https://gitlab.com/gitlab-org/gitlab-ce/issues/38242#note_41113724 "GitLab should be fine to run in Puma, IIRC people are already doing this and when I last tested Puma I didn't really run into any issues with it."

Is Puma support reliable then? Any update?

@timofonic I think Puma support is still experimental. See:

uyiyiyuiyi

lmlml

llklklkkl

kjkl
lkllk

changed the description

886	890	pg (~> 0.18.2)
887	891	poltergeist (~> 1.6.0)
Ramya Authappan @rauthappan · 6 years ago Developer uyiyiyuiyi Please register or sign in to reply

 # === Puma control rack application ===
 # Start the puma control rack application on "url". This application can
 # be communicated with to control the main server. Additionally, you can
 # provide an authentication token, so all requests to the control server
 # will need to include that token as a query parameter. This allows for
 # simple authentication.
+#
 # Check out https://github.com/puma/puma/blob/master/lib/puma/app/status.rb
 # to see what the app has available.
+#
 # activate_control_app 'unix:///var/run/pumactl.sock'
 # activate_control_app 'unix:///var/run/pumactl.sock', { auth_token: '12345' }
 # activate_control_app 'unix:///var/run/pumactl.sock', { no_token: true }
+#
 # vim: set ft=ruby:

   gem 'unicorn-worker-killer', '~> 0.4.2'
 end
 group :puma do
   gem 'puma', '~> 2.15', require: false
   gem 'puma_worker_killer', require: false
 end
 # State machine

Admin message

Admin message

locking test

Merge request reports

Activity

Refs:

Unicorn

Puma