Backup failures with different db versions

From: https://gitlab.zendesk.com/agent/tickets/81998

Customer is running database on a separate node. Database node was upgrade to GitLab 9.5.2 and PostgreSQL 9.6.3. Application node failed to upgrade due to the PostgreSQL binaries on the application node still being for 9.2.

We should document the correct upgrade path for this situation.

@ibaum Do we need to expand https://docs.gitlab.com/omnibus/settings/database.html#backup-and-restore-a-non-packaged-postgresql-database ?

That might work. In this case, the user was using a packaged PostgreSQL database, just on a different server. So perhaps if we make that section account for that we should be ok.

I need to double check, but I think newer pg_dump will work with older versions. So it may be better to advise in this case to upgrade the application server first.

I think I found the root cause of the Problem:

In https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/helpers/base_pg_helper.rb the contents of the file /var/opt/gitlab/postgresql/data/PG_VERSION are evaluated.

This contains 9.2.x as the version that placed there during the original installation. But since the Database on the GL server was not updated (there is no actual database there), it never got changed to 9.6.

But on each update, the Chef recipy will first force the symlinks for the database tools to 9.2 as it believes that this is the current version of the DB. Then it will attempt to backup the database and will fail as the versions of the tools and the server don't align.

I will manually change the contents of the file to the actual version and then reconfigure.

Personally I think that the documentation should have a separate section on a package but remote DB. I did not think that the above chapter applied to me, as my DB was package. Just on a different machine (explicitly recomended in the Gitlab-HA documentation!)

Thanks @cypherfox. I agree this belongs in a separate section. We should also include a note about the old database files you ran into.

The separate section should explicitly state on which of the two machines pg_upgrade should be run, and what further task should be done on the other.

And the upgrade script should be fixed to run without a redis (or any other service) requirement.

assigned to @ibaum

created branch 2746-backup-failures-with-different-db-versions

changed milestone to %9.5

mentioned in merge request !1916 (merged)

added customer label

closed via merge request !1916 (merged)

mentioned in commit 36c03dc4

mentioned in commit f96f5540