Skip to content

Fix flaky examples tracking

username-removed-128633 requested to merge rc/fix-flaky_example into master

EE MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3065

What does this MR do?

From the commits:

Ensure RSpecFlaky doesn't automatically update flaky examples

Previously, instantiating a RspecFlaky::FlakyExample object would automatically update its first_flaky_at, last_flaky_at and last_flaky_job (even if the example was not flaky in the current spec run!).

That was wrong because we would overwrite every time the suite report with this false data.

We now:

  • Get the suite report and only read from it
  • Write only the currently detected flaky examples in the report, so that the final report is only updated with flaky examples that were actually detected in each job. Before, job1 could overwrite the legit report from job2!
  • Write the newly detected flaky examples by rejecting the already tracked flaky specs instead of using another hash.

Are there points in the code the reviewer needs to double check?

I've improved the specs and added more specs so everything should be better.

I've changed the name of the reports so that the detection will start from scratch since we cannot trust the data we currently have in report-master.json.

Why was this MR needed?

Because, the first_flaky_at, last_flaky_at and last_flaky_job in the report-master.json file are not accurate.

Does this MR meet the acceptance criteria?

Edited by Grzegorz Bizon

Merge request reports