Skip to content

Add new --only-failures CLI option

gitlab-qa-bot requested to merge rerun-failures into master

Created by: myronmarston

(Note: this PR has changed significantly compared to when it was first opened. See this comment for current status).

This is the start of the implementation of a new --rerun-failures CLI flag that runs just the failures from the last run. This leverages the new example IDs to ensure it selects the exact examples that failed the last time. My hope with this feature is that it "just works" the first time a user decides they want to run just the failures from the last run, w/o having to configure anything in advance to enable it.

As part of the implementation, this creates a new .rspec_last_run directory in the users's home directory that persists the list of failures from the last run so we can use it on the next run. I can imagine us finding other uses for this directory in the future.

There are a lot of open questions:

  • Do we always want to write out the failures to .rspec_last_run? Specific cases I can think of where we may not want to:
    • All specs pass -- do we blow away what was there before and replace it with nothing?
    • The user uses --fail-fast -- in that case it's going to abort after one failure and the user may want to get all failures, and then use --rerun-failures --fail-fast to repeatedly run just one of the remaining failing specs...but if we replaced the persisted failures with a single --fail-fast spec it wouldn't support this workflow.
    • The user runs only one example (via a location or id filter).
  • What do we do if we can't write out the failures list to disk? Emitting an error or warning on every run would be very annoying/noisy for users who don't want to use this option.
  • What do we do if --rerun-failures is passed and there is no persisted failures from a prior run? Do we abort with an error or treat it like "all examples filtered out" and respect the run_all_when_everything_filtered config option?
  • What do we do if the user tries to pass --rerun-failures and also specifies specific spec files to load, or locations/ids to run? Should we disallow that or try to combine them somehow? For example, maybe we could take the intersection of the two? So rspec spec/models --rerun-failures would run just the failures in the model specs. This situation may also relate to whether or not we update the persisted failures every time.
  • The recorded example ids may no longer be valid or may refer to different examples than they did when they were recorded (such as when changing branches or pulling new commits). Should we try to detect this and do anything about it?

/cc @rspec/rspec

Merge request reports