Skip to content

WIP: Display link to testcase issue if test fails

James Nutt requested to merge ml-display-testcase-link-for-all-failed-tests into master

This will allow the testcase issue reporter to be used in all test pipelines.

If a test fails it will find the issue for the test (if it exists) and display a link in the job log.

That makes it easier to find out if there are known failures for a test that fails in an MR (or any other non-scheduled pipeline).

Closes https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/444

Test results

Here's what happens if the test failed in the Staging pipeline. This is the same as the previous behavior:

Reporting test results in `fixtures/stage_report_input/rspec-failure.xml` as issues in project `mlapierre-test/testcases-test` via the API at `https://gitlab.com/api/v4`.
Reporting tests in fixtures/stage_report_input/rspec-failure.xml
Reporting test name: Create Codeowners displays owners specified in CODEOWNERS file | ./qa/specs/features/ee/browser_ui/3_create/repository/code_owners_spec.rb
Found existing issue: https://gitlab.com/mlapierre-test/testcases-test/-/issues/216
Issue updated

Now here's what happens if the test failed in an MR:

Reporting test results in `fixtures/stage_report_input/rspec-failure.xml` as issues in project `mlapierre-test/testcases-test` via the API at `https://gitlab.com/api/v4`.
Reporting tests in fixtures/stage_report_input/rspec-failure.xml
Reporting test name: Create Codeowners displays owners specified in CODEOWNERS file | ./qa/specs/features/ee/browser_ui/3_create/repository/code_owners_spec.rb
Found existing issue: https://gitlab.com/mlapierre-test/testcases-test/-/issues/216

Note that it doesn't say Issue updated. But now I could click that link to find out if there are known failures (at least after the reporter is enabled for MRs. I'll do that after this is merged).

This is the output of a passing test in master (same as previous behavior):

Reporting test results in `fixtures/stage_report_input/rspec-pass.xml` as issues in project `mlapierre-test/testcases-test` via the API at `https://gitlab.com/api/v4`.
Reporting tests in fixtures/stage_report_input/rspec-pass.xml
Reporting test name: Plan Sum of issues weights on issue board shows the sum of issues weights in the board list's header | ./qa/specs/features/ee/browser_ui/2_plan/issue_boards/sum_of_issues_weights_spec.rb
Found existing issue: https://gitlab.com/mlapierre-test/testcases-test/-/issues/250
Issue updated

And a passing test in an MR:

Reporting test results in `fixtures/stage_report_input/rspec-pass.xml` as issues in project `mlapierre-test/testcases-test` via the API at `https://gitlab.com/api/v4`.
Reporting tests in fixtures/stage_report_input/rspec-pass.xml

I.e., passing tests are skipped (because we don't really need to know any more when tests pass).

Edited by James Nutt

Merge request reports