Upgrade dbs to latest gitlab-ee/postgres package
What are we going to do?
Upgrade postgres fromm 9.6.1 to 9.6.3
Why are we doing it?
Bug fixes and stability see this issue https://gitlab.com/gitlab-com/infrastructure/issues/1158 for more details
When are we going to do it?
- Start time: 16:00 UTC
- Duration: 30min
- Estimated end time: 16:30 UTC
How are we going to do it?
- stop slave dbs
- stop master db
- change chef-role (https://dev.gitlab.org/cookbooks/chef-repo/blob/master/roles/gitlab-base-db-postgres.json)
- run
chef-client
- run
gitlab-ctl reconfigure
(manually) - start master
- start slaves
How are we preparing for it?
Merge request for postgres role here: https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/839
What can we check before starting?
- All the slaves are up2date
- Stop the sidekiq/sidekiq-cluster services
What can we check afterwards to ensure that it's working?
version is the new one
Impact
- Type of impact: downtime
- What will happen: restart the core db -> everything will be stopped
- Do we expect downtime? (set the override in pagerduty): Yes
How are we communicating this to our customers?
- Tweet before and after the change.
- Do we need to set a broadcast banner?: Yes
What is the rollback plan?
Monitoring
- Graphs to check for failures:
-
- Graphs to check for improvements:
- Next time we need the specific functions on postgres
- Alerts that may trigger:
- All of them
[IF NEEDED]
Google Doc to follow during the change (remember to link in the on-call log)
Scheduling
Schedule a downtime in the production calendar twice as long as your worst duration estimate, be pessimistic (better safe than sorry)
When things go wrong (downtime or service degradation)
- Label the change issue as outage
- Perform a blameless post mortem
References
Edited by Jason Tevnan