Add manual QA on staging as a step for each significant RC

What

Adds manual Quality Assurance as a step which should be considered during each Release Candidate after deploying to staging. Previously it was required once for the overall release but not given a specific time.

Why

Easy to miss it for a given RC at the moment as we only ask for it to be completed once before the official release. Could have helped avoid https://gitlab.com/gitlab-org/gitlab-ce/issues/33999

Closes https://gitlab.com/gitlab-org/release-tools/issues/72

assigned to @ClemMakesApps

@ClemMakesApps What do you think?

I think that for the incident that happened for RC5, this would have resolved it but I fear this may add too much overhead. Honestly, the root cause for why RC5 broke merge requests was because we didn't wait for the builds to turn green before deployment.

Perhaps adding something to the checklist saying that you should not proceed to tagging until builds are green (for either .com or Dev)? That alone should have prevented that problem and adding one additional point to the checklist seems like less overhead than going through an entire tedious QA checklist that would take up an additional 30 minutes+ of time

assigned to @jamedjo

@ClemMakesApps Agreed on root cause, and think https://gitlab.com/gitlab-org/release-tools/merge_requests/145/diffs should have a bigger impact on preventing that problem. A step to check those are green makes sense. Although presumably made redundant if we use the MR process?

There are sometimes things where manual testing identifies flaws not spotted by the tests, so this could still be useful. Would it be helpful if this was a non-blocking step to help us identify regressions quicker? Or better to leave as is so it is independent of the RC process and just done once for the release?

I think it would be better to make our deploy script fail if the ci is not green yet. Add a -f flag to force deploy even though CI is still running.

That seems like a more optimal solution imo

Perhaps adding something to the checklist saying that you should not proceed to tagging until builds are green (for either .com or Dev)?

@ClemMakesApps We already have that?:

  - [ ] Ensure tests are green on [CE stable branch]
  - [ ] Ensure tests are green on [EE stable branch]
  - [ ] Ensure tests are green on [Omnibus CE stable branch]
  - [ ] Ensure tests are green on [Omnibus EE stable branch]

@jamedjo I'm not sure about the change in this MR, soon you';ll be able to run GitLab QA for an arbitrary commit/MR (https://gitlab.com/gitlab-org/gitlab-qa/issues/32).

That said, we still miss some feature coverage: https://gitlab.com/gitlab-org/gitlab-qa/issues/3

We already have that?

Yeah, but we should make it part of the deploy script as another sanity check in case a RM makes an assumption that a branch is going to pass (and then it doesn't) as seen in https://gitlab.com/gitlab-com/infrastructure/issues/2114

Oh yeah I see, we could pretty easily make an API call for the tag that's being deployed and ensure that it's green. Ideally it would check the last commit before the tag since the tagging commit itself cannot fail the pipeline, that would be a shame to wait 1h just for the tag commit to be green.

To clarify my blurb above:

Another complication for that would be that using an MR might mean the MR was green but there isn't a green badge on that page. If merged through the UI then a merge commit will have been introduced, and if merged fast-forward manually because the pipeline which has passed for that commit doesn't show on the stable branch.

Still worth exploring though