Add manual CI job to run tests against staging

It is now possible to execute tests against staging using a command line, but it would be also good to leverage CI and manual actions for that.

Extends #2 (closed)

We talked with @pcarranza and we want to implement this \cc @rymai.

Some thoughts:

1. We run GitLab QA tests against a package version deployed to staging.
2. We already did some work that makes it possible to trigger GitLab QA from Omnibus.
3. This can be a manual action, or automated task, but because our staging deployment is mostly manual, maybe we need to make GitLab QA against staging manual as well.
4. We don't need much changes in GitLab QA itself. We already support running tests against any instance of CE or EE. We need create GitLab QA user and namespace that we are going to use on staging, the most simple way is to create it on production.
5. We might need to add a few environment variables to QA code, to control a name of the GitLab QA namespace etc. but this is easy.
6. We agreed with @pcarranza that QA test suite is very limited, but running it against staging in an automate / semi-automated manner, still brings some value.

cc/ @omame @jarv @jameslopez

The idea would be that we could wire takeoff, and when we finish deploying a version in staging we spawn a GitLab-QA execution somehow.

This can provide 2 main advantages:

• We will know when we broke staging after making a change to it from the infrastructure side. We can prevent this from impacting deployments by using GitLab-QA to know that staging is still functional.
• We will know if a given deployed version survives the critical path.

I know that we still don't have the whole critical path covered, but wiring deployments to GitLab-QA will be an enabler to actually getting us investing the time into increasing the coverage of the tool, and will definitely reduce the toil of deployments.

cc/ @sitschner

We agreed with @pcarranza that QA test suite is very limited, but running it against staging in an automate / semi-automated manner, still brings some value.

If we wire https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2581 as part of the same task we can move on this item as well. The easier we make it for QA to run automatically, the less excuses there are for not adding items to QA.

It sounds like we need takeoff to trigger the execution then, then we will need to push the results to slack so they slap us in the face or something like that.

The lower the bar, the easier adoption.

As far as I remember takeoff is written in Ruby, and we also have a gitlab-qa gem that we can use as a library, so we can integrate it into takeoff and run GitLab QA directly from the place that takeoff is executed from (release manager's machine?). Only dependencies are Docker and Ruby and gitlab-qa gem. We do something similar in Omnibus already. \cc @balasankarc

mentioned in issue takeoff#17

Created https://gitlab.com/gitlab-org/takeoff/issues/17 to discuss how to integrate this with takeoff

@ayufan Would it make sense to allocate some time to work on this and to move this issue forward?

@grzesiek As a first iteration, couldn't we just trigger a pipeline (that would run QA against staging) in gitlab-org/gitlab-qa once a deploy to staging is done (i.e. trigger the pipeline at the end of takeoff deploy command)?

@rymai I think that we need a few changes before we can do that, but this is a good plan indeed.

1. Add a gitlab-qa user on production (so that is available on staging as well).
2. Add a gitlab-qa test namespace. All test namespaces are going to be created there as a subgroups. This needs some changes in in-project QA and QA coordinator as well.
3. Prepare a manual job that we would be able to play using a takeoff.

@grzesiek

• Add a gitlab-qa user on production (so that is available on staging as well).

Created https://gitlab.com/gitlab-qa!

• Add a gitlab-qa test namespace. All test namespaces are going to be created there as a subgroups. This needs some changes in in-project QA and QA coordinator as well.

I guess the gitlab-qa user namespace is enough?

Prepare a manual job that we would be able to play using a takeoff.

I will take a stab at this! I guess the only missing part will be to sync production database to staging.

• Add a gitlab-qa test namespace. All test namespaces are going to be created there as a subgroups. This needs some changes in in-project QA and QA coordinator as well.

I guess the gitlab-qa user namespace is enough?

@rymai I mean that project-level QA should create test namespace (per-test run) as a subgroup. Currently these are top-level groups.

@rymai I mean that project-level QA should create test namespace (per-test run) as a subgroup. Currently these are top-level groups.

@grzesiek Ok, sounds good!

@grzesiek I've created a gitlab-qa-sandbox group while logged-in as the gitlab-qa user on production.

I will create an issue in CE to update the QA steps to create a subgroup of this group instead of a top-level group.

@rymai Awesome! Please note that we might want to pass top-level group name using an environment variable, and create one if environment variable is not set to make it work with any instance.

For cross-reference: gitlab-org/gitlab-ce#37996

@grzesiek @rspeicher FYI will help implementing this soon.

assigned to @rspeicher

Awesome, thanks!

mentioned in commit 14ea21c5