pgbouncer from omnibus with consul

This is the last piece of the HA postgrers setup.

pgbouncer should watch the postgres service and configure the proxy accordingly.

mentioned in issue #2171

mentioned in issue #2173 (closed)

Some work has already been done:

• https://gitlab.com/gitlab-com/infrastructure/issues/2173#note_36805064
• https://gitlab.com/gitlab-com/infrastructure/issues/2173#note_36804660

I was having some problems getting a watcher to watch this service though: This watcher:

{
  "watches": [
  {
    "type": "service",
    "service": "postgresql",
    "handler": "/usr/local/bin/update_pgbouncer.rb"
  }
  ]
}

should trigger this script:

#!/opt/gitlab/embedded/bin/ruby

require 'json'

data = $stdin.gets

struct = JSON.parse(data)

new_master = ''

struct.each do |entry|
  entry['Checks'].each do |check|
    if check['CheckID'].eql?('service:postgresql') && check['Status'].eql?('passing')
      new_master = entry['Node']['Address']
    end
  end
end

ini_file="/var/opt/gitlab/pgbouncer/pgbouncer.ini"
`erb host=#{new_master} #{ini_file}.erb > #{ini_file}`
`pkill -1 pgbouncer`

where the erb file: /var/opt/gitlab/pgbouncer/pgbouncer.ini.erb contains a place holder for the ip/hostname of the master.

If i trigger this watcher manually: consul watch -type=service -service=postgresql /usr/local/bin/update_pgbouncer.rb it works fine, but via the configured watcher in the /etc/consul/conf.d/pgbouncer.json it is not being triggered.

@ibaum is hard at work on the packaged version, which might solve my issue. I will also be out next week so when i come back he will already have solved ALL my problems :)

added goal label

removed goal label

Removing goal because @jtevnan is out this week.

removed milestone

mentioned in issue #1485 (closed)

just for reference: there was talk of service discovery 4 months ago already here https://gitlab.com/gitlab-com/infrastructure/issues/1485

changed milestone to %WoW ending 2017-08-22

as an FYI we need https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2628 before we can move over to Omnibus since the current omnibus pgbouncer doesn't allow setting custom pool sizes.

staging does not have this limitation, we just need to have it ready before we go to prod with it, but that is duly noted.

/cc @marin

After discussing with @ibaum and @marin we have the following plan:

• get the omnibus version running on staging from a custom package from this mr: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1800
• @marin is confident that this will be merged into the upcoming release if there are no red flags
• @ibaum will also finish up a change for the issue @yorickpeterse mentioned (https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2628) so we will not have any production blockers.
• let it run in staging until the middle of next week to ensure we have no other issues.
• roll it out to production mid-week
• since it is a deploy week, the 21/22nd are out of the question, leaving the 23/4th for the change.
• while we dont expect anything to happen, we should will downtime.

mentioned in issue #2168 (closed)

mentioned in issue #2463 (closed)

https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/1045

https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/1050

mentioned in issue #2581

closed

Closing this issue since we have to take downtime to get this live in prod: https://gitlab.com/gitlab-com/infrastructure/issues/2581