Skip to content

Relate existing failure issues upon test failures or create new ones

George Koltsov requested to merge find-failure-issue-for-test-failure into master

This adds a new --relate-failure-issue option to exe/gitlab-qa-report that does a few things:

  1. Detect existing QA failure issue in the $QA_FAILURES_REPORTING_PROJECT project (https://gitlab.com/rymai/gitlab-qa-issues/ is used in this MR for testing purpose)
    • The detection is done in two steps:
      1. Detect issues with a title that includes the test path or description
      2. Among the potential issues, select the one for which the stacktrace is the closest to the failed test's one (a 5% diff margin is allowed).
  2. If no issue could be detected, this will create a new issue (e.g. https://gitlab.com/rymai/gitlab-qa-issues/-/issues/13)
  3. It will mark the QA failure issue as related to the testcase issue (e.g. https://gitlab.com/rymai/gitlab-qa-issues/-/issues/13).

Note:

  1. Before we enable this globally, we can define the QA_FAILURES_REPORTER_OPTIONS: "--dry-run" variable to only print a message instead of creating/updating issues.
  2. A --max-diff-ratio option is available to fine-rune the threshold above which we don't consider two stacktraces similar (this can be set as part of QA_FAILURES_REPORTER_OPTIONS).

Related to https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/586.

Edited by George Koltsov

Merge request reports