Test repmgrd in staging

After a successful test of repmgr (https://gitlab.com/gitlab-com/infrastructure/issues/1959) we would like to test the automation part repmgrd as @ibaum mentioned here: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1807#note_32146199

I left a step out of my notes, needed to set postgresql['shared_preload_libraries'] = 'repmgr_funcs' on the database nodes

we originally had: db04 - master db05 - slave db06 - slave

we stopped db04 and waited for repmgrd to respond.

withing 60 seconds it promoted db05:

[2017-06-13 20:25:31] [NOTICE] promoting standby
[2017-06-13 20:25:31] [NOTICE] promoting server using '/opt/gitlab/embedded/bin/pg_ctl -D /var/opt/gitlab/postgresql/data promote'
[2017-06-13 20:25:33] [NOTICE] STANDBY PROMOTE successful

and made 06 a follower (automatically):

[2017-06-13 20:25:36] [NOTICE] restarting server using '/opt/gitlab/embedded/bin/pg_ctl  -w -D /var/opt/gitlab/postgresql/data -m fast restart'
[2017-06-13 20:25:38] [NOTICE] STANDBY FOLLOW successful
[2017-06-13 20:25:38] [NOTICE] node 3 now following new upstream node 2

To sync db04 up again we first started repmgrd hoping that it would trigger correct actions, however since it looks at the same database it is meant to sync, this lead the WAL locations to run adrift (as @ibaum mentioned)

So in our case at the moment, we have to - recover old master, make it a following member and THEN start repmgrd

closed

mentioned in issue #1959 (closed)