GitLab Pages - Outage 10:15 PM UTC 2017/03/10

Overview

1. GitLab pages HTTPS redirects fail in an infinite loop - 10:15 PM UTC 2017/03/10

2. @yorickpeterse mentions this on Slack https://gitlab.slack.com/archives/infrastructure/p1489184308193924

3. @stanhu mentions a fix for the configuration /opt/gitlab/sv/gitlab-pages/run - https://gitlab.slack.com/archives/infrastructure/p1489185831468207 - - 10:45 PM UTC 2017/03/10

   # diff run.orig run
   27,28c27,28
   <     -redirect-http=false \
   <     -use-http2=true \
   ---
   >     -redirect-http false \
   >     -use-http2 true \

4. This fix is deployed but the problem persists.

5. Stan mentions an issue with app.go if https && !a.RedirectHTTP - https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1348#note_25156832 - 10:53 PM UTC 2017/03/10

6. This is rolled into a new diff patch https://gitlab.com/snippets/1650007/raw - 1:35 AM UTC 2017/03/10

7. Patch deployed - 1:45 AM UTC 2017/03/10

8. Problem persists. Investigation reveals gitlab-pages is refusing to restart with the standard gitlab-ctl restart command - 2:00 AM UTC 2017/03/10

9. GitLab pages is force restarted sudo kill -9 $(ps aux | grep '[g]itlab-pages-unprivileged' | awk '{print $2}') && sudo gitlab-ctl start gitlab-pages" -t 5 - 2:20 AM UTC 2017/03/10 - Problem resolved.

---

Problems

1. Incorrect conditional statement added 1 year ago https://gitlab.com/gitlab-org/gitlab-pages/commit/3d691bf42432304a3ae6d6465c3f596629891e74

2. Not sure how this changed on production as the config for cluster and the new web works was http redirect = false

cc @stanhu @pcarranza @omame

changed title from GitLab Pages to GitLab Pages - Outage 10:15 PM UTC 2017/03/10

I might be wrong, but when I looked some time after step 4, the fix got reverted.

Proposed fix for GitLab Pages: gitlab-org/gitlab-pages!21 (merged)

assigned to @maratkalibek

removed assignee

closed