Failure filtering runs all examples when there are no failures
Created by: myronmarston
When you have config.run_all_when_everything_filtered = true
set and you use --only-failures
or --next-failure
and there are no failures in the persistence file, the current behavior is to run all examples. This is counterintuitive because you've told RSpec to run only failures but instead it runs only successes (which happens to be all examples). This is a consequence of the fact that failure filtering is implemented using the general filtering mechanism, and run_all_when_everything_filtered
kicks in in this case.
Ideally, if you used --only-failures
or --next-failure
and there were none, RSpec would abort quickly with a message stating there are no failures. If we do that, one open question is what status code to use: on the one hand, the command kinda failed (it failed to find any failures to run) so maybe it should be non-zero. Also, exiting with a zero status code when no examples were even run seems potentially problematic. OTOH, I could see someone repeatedly running rspec --next-failure && say "done"
(or similar -- idea inspired by this tweet), and for this use case it would be important for RSpec to exit with a 0 status when it hits a point where there are no more failures to run.
Thoughts on what status code should be used for this?
We're also discussing phasing out run_all_when_everything_filtered
in #1920 (closed) in favor of a new ignore_unmatched_filters
option, so that should be taken into account for whatever solution we decide upon.
/cc @rspec/rspec