`rspec-retry` is bitting us when some API specs fail
We've been bitten in the past with a tricky bug that would happen only on CI:
Imagine we have the following spec:
get api("/projects/#{project.id}", admin)
expect(json_response['id']).to eq(project.id)
expect(json_response['builds_enabled']).to be_present
Let's imagine json_response['id'] == 1
here.
- The API spec fails for the first time, for instance on
expect(json_response['builds_enabled']).to be_present
because the field is nowjobs_response
. This is a legit failure, and that's the one we should see at the end of the CI run. -
rspec-retry
kicks-in (because we setRSPEC_RETRY_RETRY_COUNT: "3"
in our.gitlab-ci.yml
) - Unfortunately,
rspec-retry
doesn't rerun the example with a clean slate (it only clearlet
variables), meaning that ourjson_response
method will still return the memoized response from the first run, including the ID of the resource from the first run (i.e.json_response['id'] == 1
)... - Thus, on the second run of the spec,
expect(json_response['id']).to eq(project.id)
will fail becauseproject
which waslet
was cleared, thus we have a new record, with an ID of2
so our expectation is nowexpect(1).to eq(2)
=> FAIL -
rspec-retry
retries two more times, before failing with the following failure:
Failure/Error: expect(json_response['id']).to eq(project.id)
expected: 2
got: 1
(compared using ==)
At this point you're trying to understand what's going on.
Anyways, the easiest solution to this weird bug is to implement json_response
as a let
variable so that it's cleared on each retries.