Document need for tuning default_pool_size and max_client_conn

https://gitlab.zendesk.com/agent/tickets/81938

Customer had a 2-node GitLab setup and tried to set up pgbouncer outside of omnibus. The default pool size of 20 does not appear to be enough, since the default GitLab Omnibus setup uses:

• 10 connections for Rails
• 5 (?) for Sidekiq

By default, default_pool_size is set to 20. If this is set too low, we see errors such as:

/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:651: warning: Failed to set the default_internal encoding to UTF8: 'ERROR: query_wait_timeout 
server closed the connection unexpectedly 
This probably means the server terminated abnormally 
before or while processing the request. 
' 
rake aborted! 
PG::Error: invalid encoding name: unicode 
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/ignorable_column.rb:17:in `columns' 
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting.rb:221:in `ensure_cache_setup' 
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting.rb:214:in `cached' 
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:27:in `cached_application_settings' 
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:22:in `ensure_application_settings!' 
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:7:in `current_application_settings' 
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:21:in `settings' 
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:11:in `influx_metrics_enabled?' 
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics.rb:7:in `enabled?' 
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/8_metrics.rb:158:in `<top (required)>' 
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>' 
/opt/gitlab/embedded/bin/bundle:22:in `load' 
/opt/gitlab/embedded/bin/bundle:22:in `<main>' 
Tasks: TOP => gitlab:backup:create => environment 
(See full trace by running task with --trace)

In the pgbouncer logs, it shows up as timeouts because connections time out after inactivity:

2017-08-31 15:32:12.212 2275 LOG C-0x185d8f0: gitlabhq_production/gitlab@10.163.6.140:33138 closing because: query_wait_timeout (age=120)
2017-08-31 15:32:12.212 2275 WARNING C-0x185d8f0: gitlabhq_production/gitlab@10.163.6.140:33138 Pooler Error: query_wait_timeout

This happened in staging (https://gitlab.com/gitlab-com/infrastructure/issues/2565#note_38002255), so I think we need to at least make this searchable in the documentation.

/cc: @dblessing, @lbot, @ibaum

@ibaum We should change this number anyway too, we don't use that value on gitlab.com. Should we just set the number we are using there and ship it like that?

changed milestone to %10.0

added Maintenance label

I believe we're using 1000 on gitlab.com, which may be a bit high for a default value. But a higher default isn't unreasonable, as well as improving the documentation.

marked this issue as related to #2649