Bisect
Created by: myronmarston
This adds a new --bisect
flag for #1767 (closed).
TODOs:
-
Running a DRb server opens the process up to undesirable remote communication. We should probably use drb/acl to limit access to just localhost. -
Sometimes users use ENV vars to toggle certain behaviors in their test suite, and we need to make sure that any passed ENV vars get forwarded from the rspec --bisect
process to the test suite runs. -
If there are any puts
orwarn
statements in the test suite, it will print output in the middle of the bisect progress output. We should silence it. -
If the test suite bails with an error while booting (e.g. due to a NoMethodError
in adescribe
block or whatever), we don't surface that to the user. -
Bisect only works if the ordering is consistent. If the user is using a custom ordering that is inconsistent (such as using a shuffle
in there...), we should detect that and bail with a clear error about the problem. -
The runner ignores --format progress
when running the suite but not-fp
. This should be fixed. -
Ensure things work OK when no examples fail. -
Ignore additional failures that didn't fail the on the first run. -
Figure out why the AppVeyor build for 2.1 is failing. -
When bisection fails (e.g. due to bailing early from a spec suite load time error, or inconsistent ordering or whatever) we should exit with a non-zero status. -
Handle SIGINT, so that it prints the most minimal repro command it's been able to find so far and then exits -- that way, when bisect is taking along time, if the user gets impatient and hits ctrl-c
, they can still benefit from the work we've done up to that point.