Test session report should handle its own failed jobs
We're currently relying on the test reports in the form of artifacts. However, the job itself might fail in some way, like timing out or failed to upload artifacts, or any other potential issues which prevent it from uploading the artifacts.
This means we still need to iterate through all the QA jobs to make sure they fail in a way that we do have the correct artifacts attached.
- If a job is allowed to fail, indicate that as well
The following discussion from !526 (merged) should be addressed:
-
@godfat-gitlab started a discussion: (+2 comments) There's an issue here. A job timed out: https://gitlab.com/gitlab-org/gitlab-qa/-/jobs/788740090
And because of that it did not generate a report (no artifacts) and the test session wasn't aware of that thus we ended up with 0 failures in https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues/8
We probably cannot fully rely on the artifacts, because they can be missing. We still need to iterate through the jobs somehow.