Setup repmgrd in staging with latest package

Using the latest package, configure repmgrd to automatically failover db with the omnibus package.

• check in with @ibaum to figure out what else we need
• MR for omnibus cookbook to allow repmgr items into the gitlab.rb

mentioned in issue #2171

changed milestone to %WoW ending 2017-07-11

added goal label

https://gitlab.com/gitlab-org/cookbook-omnibus-gitlab/merge_requests/52 for the repmgr item in gitlab.rb

changed the description

marked the checklist item MR for omnibus cookbook to allow repmgr items into the gitlab.rb as completed

    +repmgr['node_number'] = 2
    +repmgr['enable'] = true
    +repmgr['trust_auth_cidr_addresses'] = ["10.123.2.0/22"]
    +repmgr['user'] = "repmgr"
    +repmgr['database'] = "repmgr"

after https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/859 the gitlab.rb looks good for our setup

FYI: @ibaum

changed milestone to %WoW ending 2017-07-18

added moved 1 label

added blocked label

blocked until we can get 9.4.RC3 (or 4) out the door with these changes: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1733

changed the description

changed milestone to %WoW ending 2017-07-25

added moved 2 label

changed milestone to %WoW ending 2017-08-01

now that this decision has been made to go on with consul (https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2571#note_35898366) we can proceed to configuring repmgrd accordingly.

@ibaum will have a preliminary package ready at the begining of next week.

We should get a consul cluster up in staging though since getting that side of things into the package may take longer.

mentioned in issue #2352 (closed)

@jtevnan are we still blocked in this issue?

added moved 3 label

changed milestone to %WoW ending 2017-08-08

The decision has been made to go on with consul (gitlab-org/omnibus-gitlab#2571 (comment 35898366)) For repmgrd this means:

• we need a consul cluster in staging (https://gitlab.com/gitlab-com/infrastructure/issues/2352)
• the respective consul agents have to be configured for the cluster on the db servers.
• @ibaum will have/has a preliminary package ready for us to manually configure repmgd to use consul.
• pgbouncer has to be installed and configured. (something i can do now) (https://gitlab.com/gitlab-org/cookbook-omnibus-gitlab/merge_requests/53/commits)

marked this issue as related to #2352 (closed)

@jtevnan as of https://gitlab.com/gitlab-com/infrastructure/issues/2352 being closed, not blocked anymore?

removed blocked moved 3 labels

Yes: i unblocked this issue: Next steps:

• ensuring pgbouncer is configured correctly
• ensuring consul agent is configured correctly
• configuring them all to play together with @ibaum

added moved 3 label

pgbouncer=# show clients;
 type |   user    |      database       | state  |     addr     | port  |  local_addr  | local_port |    connect_time     |    request_time     |    ptr    |   link    | remote_pid | tls
------+-----------+---------------------+--------+--------------+-------+--------------+------------+---------------------+---------------------+-----------+-----------+------------+-----
 C    | gitlab    | gitlabhq_production | active | 10.132.6.151 | 53922 | 10.129.1.102 |       6432 | 2017-08-03 20:46:44 | 2017-08-03 20:46:58 | 0x1b5e980 | 0x1b634a0 |          0 |

We can connect to the pgbouncer installation as a client.

waiting on https://gitlab.com/gitlab-com/infrastructure/issues/2416 to be able to configure consul-agent

Current state:

• agents are working
• service is configured (like this: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2571#note_35048343)
• the service checks (to tell who is master) are now causing problems

Since all interaction with the db or repmgr are regulated through gitalb-ctl or gitlab-psql the user we are running consul as (consul) has to interact with them.

on top of that, the way consul agent handles check execution, the user needs a shell (see: here for further info). So i created https://gitlab.com/gitlab-cookbooks/gitlab_consul/merge_requests/3

service definition: /etc/consul/conf.d/postgres.json

{
  "service": {
    "name": "postgresql",
    "tags": ["gitlab"],
    "port": 5432,
    "check": {
      "script": "/usr/local/bin/check_postgres.sh",
      "interval": "10s"
    }
  }
}

check: /usr/local/bin/check_postgres.sh (note absolute paths to embedded repmgr)

#!/bin/bash

set -e

cluster=$(/opt/gitlab/embedded/bin/repmgr  -f /var/opt/gitlab/postgresql/repmgr.conf cluster show || exit 2)
echo "${cluster}" | awk -v host=$(hostname -f) '
  $2 ~ /master/ {
  if($4 == host)
     exit 0;
  else
     exit 3;
}
'

Another feature which needs to be enabled for this to work is the setting: enable_script_checks which is falsely in the master of the cookbook, while it is not in the tagged branch which corresponds to master... (https://github.com/johnbellone/consul-cookbook/issues/463)

For the script check, I don't think it's unreasonable to give the consul user read only access to the repmgr database. With that, repmgr commands that only need to read should work.

I forgot that repmgrd does not NEED consul: it configures slaves to follow the new master itself... only pgbouncer needs consul to know who to follow. (https://gitlab.com/gitlab-com/infrastructure/issues/2422)

closed

mentioned in issue #2422 (closed)