Merge branch '2746-backup-failures-with-different-db-versions' into 'master'

Resolve "Backup failures with different db versions"

Closes #2746

See merge request !1916

doc/settings/database.md

@@ -303,6 +303,16 @@ you can remove the old database with:
 sudo rm -rf /var/opt/gitlab/postgresql/data.9.2.18
 ```
+### Upgrading a GitLab HA cluster
+If you have setup your GitLab instance per the [GitLab HA documentation](https://docs.gitlab.com/ee/administration/high_availability/database.html), upgrade the database server last. It should not be necessary to perform any other extra steps.
+
+You do not need to run `pg-upgrade` on any node besides the database node, but they should be updated to the latest version of GitLab before the database is updated.
+
+#### Troubleshooting upgrades in an HA cluster
+
+* If at some point, the bundled PostgreSQL had been running on a node before upgrading to an HA setup, the old data directory may remain. This will cause `gitlab-ctl reconfigure` to downgrade the version of the PostgreSQL utilities it uses on that node. Move (or remove) the directory to prevent this:
+  * `mv /var/opt/gitlab/postgresql/data/ /var/opt/gitlab/postgresql/data.$(date +%s)`
+
 ## Downgrade packaged PostgreSQL server
 As of GitLab 9.0, the default version of PostgreSQL is 9.6.1, but 9.2.18 is still shipped in the package.