Automate repmgrd configuration

This is being done in !1733 (merged)

mentioned in issue #1807 (closed)

mentioned in merge request !1733 (merged)

Updated my docker-compose.yml to work with repmgr. After bringing this online, there are still a couple of manual steps:

• To get the cluster working run:

  $ docker-compose exec db_one gitlab-ctl repmgr standby setup db_one -w
  $ docker-compose exec db_two gitlab-ctl repmgr standby setup db_one -w

• To get the application working run:

  $ docker-compose exec db_one gitlab-psql -d gitlabhq_production -c 'create extension pg_trgm'
  $ docker-compose exec app gitlab-rake db:migrate

To fix:

• pg_trgm extension should be fixed by #2501
• reconfigure tries to recreate pgbouncer nodes if postgresql['pgbouncer_user'] is set. This fails since its a standby node.
• Manual migration shouldn't be necessary

Testing out https://github.com/2ndQuadrant/repmgr/blob/master/docs/repmgrd-node-fencing.md

1. Create ssh keys on the db nodes
2. Running in docker node, so needed to add gitlab-psql user to AllowedUsers in /assets/sshd_config
3. Create home directory for gitlab-psql on app node
4. Authorize db node ssh keys on app node
5. Modify pgbouncer.ini to include another ini file containing only the [databases] section.
6. Add promote.sh to the image

• TL/DR: The script promotes the new master, then notifies the pgbouncer nodes of the new master by generating a new database stanza for the pgbouncer nodes, then sends a reload command to the pgbouncer node.

1. Update promote_command in repmgr.conf to use the script
2. Need to debug the script a little better(mostly it's use of psql), but it is executed by the node that was chosen as the new master. Manually executing the failed steps, somewhat corrected, was successful.

Pros of this approach:

• This is all pretty straightforward to automate
• No new additions to the package

Cons of this approach:

• Each db node needs to know about each pgbouncer node. So adding/removing pgbouncer nodes will need to propagate to the database nodes.
• the new master database node needs to execute a command on each of the pgbouncer nodes when it's promoted. We'll need to account for the possibility that a pgbouncer node is unreachable during failover.

mentioned in issue #2571

closed via merge request !1733 (merged)

mentioned in commit b3017887

mentioned in commit 5e07c576