Documentation to upgrade existing Geo installations

We don't have a documentation about how to upgrade an existing Geo installations.

The steps are simple:

• log into each node (primary and secondaries)
• Upgrade GitLab
• Test primary and secondary node, and check version in each.
• That's it.

Even if the steps are dead simple, we should mention it on https://docs.gitlab.com/ee/gitlab-geo/README.html.

@SeanPackham @axil I know it's late minute - can we have that for 9.0?

@regisF actually I wanted to ask about this as there is no info! Have you tested those steps? I didn't know it was that easy :)

changed milestone to %9.0

@axil yes I tried them. And it's that simple. However, for 9.0, I just learned that we will need a manual step on the secondary step, that I don't exactly know. So if the base could be there, I'd update it with the correct content.

Is there a preference to what node (primary, secondary) to update first? Will there be issues with the replication or data sync?

I added the preliminary info in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1431. I'm not sure whether we should split this to omnibus and source as well. We better link to the update docs instead.

mentioned in merge request !1431 (merged)

Now that I've tried to upgrade from 8.17 to 9.0, here are my thoughts:

GitLab 9.0 forces both the primary and the secondary to upgrade to PostgreSQL 9.6. This is tricky because right now because the jump from PostgreSQL 9.2 to 9.6 is considered a major revision, and the streaming replication between the two versions is not compatible. GitLab Geo 9.0 is also not compatible with 8.x.

Unless we make changes in omnibus to make it easier, users upgrading from 8.17 to 9.0 will have to:

1. Shut down the secondary
2. Upgrade the primary to GitLab 9.0 and PostgreSQL 9.6
3. Back up the /var/opt/gitlab/postgresql/data/recovery.conf on the secondary
4. Upgrade the secondary to GitLab 9.0 and PostgreSQL 9.6.
5. Rerun the replication step 3 of the Geo secondary docs

Other notes:

• On the secondary nodes, an admin must add geo_secondary_role['enable'] = true to /etc/gitlab/gitlab.rb.
• GitLab Geo 9.0 requires both nodes to be time synched to within 60 seconds of each other. This can easily be done via NTP.
• The /admin/geo status page may help diagnosing some errors
• Document the Geo rake task

/cc: @brodock

GitLab Geo 9.0 requires both nodes to be time synched to within 60 seconds of each other. This can easily be done via NTP.

@stanhu Do they have to be in the same timezone or are we considering UTC here?

@regisF could we get some assistance from the developers who worked on Geo?

@stanhu thank you is that all the information that is needed for upgrading Geo installs?

@brodock I believe the JWT token time comparison will be done in UTC irrespective to local time zone configuration. If one node is configured for USA East and the other for CET, then the comparison will work fine.

@SeanPackham Someone should verify my instructions are correct. I nominate @brodock. :)

mentioned in issue #1747 (closed)

mentioned in issue gitlab-com/support#449 (closed)

closed via merge request !1431 (merged)

mentioned in commit 39ece680

mentioned in issue #1963 (closed)

mentioned in merge request !1478 (merged)

Should the primary node be upgraded before secondary nodes or does it not matter?

@bbodenmiller you usually upgrade primary as you need to run database migrations. If you upgrade secondary first you need to restart after upgrading primary and things may fail because of database changes until the replication catches up in the database and the application is restarted

@brodock Can you add that to the documentation?

reopened

assigned to @brodock

changed milestone to %9.5

changed milestone to %10.0

Looks like this is done? https://docs.gitlab.com/ee/gitlab-geo/updating_the_geo_nodes.html

Feel free to reopen if not!

closed