Fix flaky examples tracking
EE MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3065
What does this MR do?
From the commits:
Ensure RSpecFlaky doesn't automatically update flaky examples
Previously, instantiating a
RspecFlaky::FlakyExample
object would automatically update itsfirst_flaky_at
,last_flaky_at
andlast_flaky_job
(even if the example was not flaky in the current spec run!).That was wrong because we would overwrite every time the suite report with this false data.
We now:
- Get the suite report and only read from it
- Write only the currently detected flaky examples in the report, so that the final report is only updated with flaky examples that were actually detected in each job. Before,
job1
could overwrite the legit report fromjob2
!- Write the newly detected flaky examples by rejecting the already tracked flaky specs instead of using another hash.
Are there points in the code the reviewer needs to double check?
I've improved the specs and added more specs so everything should be better.
I've changed the name of the reports so that the detection will start from scratch since we cannot trust the data we currently have in report-master.json
.
Why was this MR needed?
Because, the first_flaky_at
, last_flaky_at
and last_flaky_job
in the report-master.json
file are not accurate.
Does this MR meet the acceptance criteria?
-
Tests added for this feature/bug - Review
-
Has been reviewed by Backend
-
-
Conform by the merge request performance guides -
Conform by the style guides -
Squashed related commits together
Merge request reports
Activity
assigned to @grzesiek
- Resolved by username-removed-128633
@rymai I left just a single question! I think this is getting quite complex, and we can benefit from thinking about extracting some classes.
assigned to @rymai
assigned to @grzesiek
@rymai
lib/rspec_flaky/flaky_examples_collection.rb:5:1: C: Layout/EmptyLinesAroundClassBody: Extra empty line detected at class body beginning.
added 1 commit
- da5aa64f - Add new RSpecFlaky::FlakyExamplesCollection and RSpecFlaky::Config classes
added 1 commit
- da5aa64f - Add new RSpecFlaky::FlakyExamplesCollection and RSpecFlaky::Config classes
mentioned in commit 05054803
Thanks @rymai! Let's try it!