Skip to content

Add error-exit-code to differentiate from failures

gitlab-qa-bot requested to merge github/fork/robotdana/error_exit_code into main

Created by: robotdana

it can be helpful to know if RSpec fails because of an example or if it errors out because it couldn't load a spec file, or there was an issue in the a before(:suite) hook or etc

i've named the setting error-exit-code, and tried to add it to all the relevant places, but i don't know rspec codebase well so there was some guessing. in particular i'm not sure what bisect should do.

i have it fall back to the failure-exit-code before defaulting to 1 so there's no changes if people don't opt in to the setting.

the specific usecase of this is our CI automatically retries failures using the persistence file, and assumes if they pass on that retry they were flaky. however, the persistence file isn't written to when there's an error outside of examples, so this could mean falsely passing builds.

by checking for a different exit code, and then not running the retry we can avoid this issue.

This PR also includes me trying to make travis builds work. there were gem/ruby version issues with simplecov-html, childprocecss, & rake (these issues were also present in other RSpec projects, so when it tries to build e.g. rspec-expectations it still fails. that's when i gave up trying to make travis work)

Merge request reports