Repmgrd not working on an old master it became a standby.

Related to https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2571

After https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1800

Repmgrd was not working on an old master after we converted it to a standby.

I've observed this few times more. There are no log outputs and repmgrd seems to be active in processes but it doesn't do anything.

/cc @ibaum

mentioned in merge request !1800 (merged)

I checked the source, and it looks like repmgrd checks whether it is a master or standby node on startup, then remembers forever. Even a HUP won't make it re-evaluate.

I think we should probably update the documentation to perform a restart when converting a failed master to standby.

assigned to @ibaum

changed milestone to %9.5

added Documentation Pick into Stable labels

added postgresql label

@marin It looks like I already include the note to restart repmgrd after converting an old master to standby. Do you know if this is something you did in your testing?

marked this issue as related to #2571

@ibaum I didn't do anything manually for this. I was simulating a case where a master node becomes unavailable, automated failover gets triggered and after some time previous master node comes back online on its own. This can be a pretty common case in case of network weather.

@marin In that case, #2697 should be the proper fix.

I believe this has been addressed. We've documented that repmgrd needs to be restarted when a master node is converted to standby manually. And there is a separate issue for when a failed master comes back on its own.

closed