Makes sanity job check match on expected output instead of string comparison
What does this MR do and why?
Currently
When the sanity-framework
job runs, GitLab QA expects it to output the number of examples that ran/failed
This causes issues with adding new tests to sanity-framework
as the expectation is hard coded
This MR
Changes the expectation to only match the number of failures.
This change should allow the sanity check to expect that a test failed, but doesn't have awareness of how many are passing.
Screenshots or screen recordings
These are strongly recommended to assist reviewers and reduce the time to merge your change.
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.