Skip to content
Snippets Groups Projects

Fix flaky examples tracking

Merged username-removed-128633 requested to merge rc/fix-flaky_example into master

EE MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3065

What does this MR do?

From the commits:

Ensure RSpecFlaky doesn't automatically update flaky examples

Previously, instantiating a RspecFlaky::FlakyExample object would automatically update its first_flaky_at, last_flaky_at and last_flaky_job (even if the example was not flaky in the current spec run!).

That was wrong because we would overwrite every time the suite report with this false data.

We now:

  • Get the suite report and only read from it
  • Write only the currently detected flaky examples in the report, so that the final report is only updated with flaky examples that were actually detected in each job. Before, job1 could overwrite the legit report from job2!
  • Write the newly detected flaky examples by rejecting the already tracked flaky specs instead of using another hash.

Are there points in the code the reviewer needs to double check?

I've improved the specs and added more specs so everything should be better.

I've changed the name of the reports so that the detection will start from scratch since we cannot trust the data we currently have in report-master.json.

Why was this MR needed?

Because, the first_flaky_at, last_flaky_at and last_flaky_job in the report-master.json file are not accurate.

Does this MR meet the acceptance criteria?

Edited by Grzegorz Bizon

Merge request reports

Pipeline #12514102 passed with warnings

Pipeline passed with warnings for da5aa64f on rc/fix-flaky_example

Test coverage 72.24% (0.00%) from 2 jobs

Merged by Grzegorz BizonGrzegorz Bizon 7 years ago (Oct 6, 2017 11:40am UTC)

Loading

Pipeline #12517626 failed

Pipeline failed for 05054803 on master

Test coverage 50.91% (0.00%) from 2 jobs

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading