Relate existing failure issues upon test failures or create new ones
This adds a new --relate-failure-issue
option to exe/gitlab-qa-report
that does a few things:
- Detect existing QA failure issue in the
$QA_FAILURES_REPORTING_PROJECT
project (https://gitlab.com/rymai/gitlab-qa-issues/ is used in this MR for testing purpose)- The detection is done in two steps:
- Detect issues with a title that includes the test path or description
- Among the potential issues, select the one for which the stacktrace is the closest to the failed test's one (a 5% diff margin is allowed).
- The detection is done in two steps:
- If no issue could be detected, this will create a new issue (e.g. https://gitlab.com/rymai/gitlab-qa-issues/-/issues/13)
- It will mark the QA failure issue as related to the testcase issue (e.g. https://gitlab.com/rymai/gitlab-qa-issues/-/issues/13).
Note:
- Before we enable this globally, we can define the
QA_FAILURES_REPORTER_OPTIONS: "--dry-run"
variable to only print a message instead of creating/updating issues. - A
--max-diff-ratio
option is available to fine-rune the threshold above which we don't consider two stacktraces similar (this can be set as part ofQA_FAILURES_REPORTER_OPTIONS
).
Related to https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/586.
Edited by George Koltsov