Skip to content

Bisect

gitlab-qa-bot requested to merge bisect into master

Created by: myronmarston

This adds a new --bisect flag for #1767 (closed).

TODOs:

  • Running a DRb server opens the process up to undesirable remote communication. We should probably use drb/acl to limit access to just localhost.
  • Sometimes users use ENV vars to toggle certain behaviors in their test suite, and we need to make sure that any passed ENV vars get forwarded from the rspec --bisect process to the test suite runs.
  • If there are any puts or warn statements in the test suite, it will print output in the middle of the bisect progress output. We should silence it.
  • If the test suite bails with an error while booting (e.g. due to a NoMethodError in a describe block or whatever), we don't surface that to the user.
  • Bisect only works if the ordering is consistent. If the user is using a custom ordering that is inconsistent (such as using a shuffle in there...), we should detect that and bail with a clear error about the problem.
  • The runner ignores --format progress when running the suite but not -fp. This should be fixed.
  • Ensure things work OK when no examples fail.
  • Ignore additional failures that didn't fail the on the first run.
  • Figure out why the AppVeyor build for 2.1 is failing.
  • When bisection fails (e.g. due to bailing early from a spec suite load time error, or inconsistent ordering or whatever) we should exit with a non-zero status.
  • Handle SIGINT, so that it prints the most minimal repro command it's been able to find so far and then exits -- that way, when bisect is taking along time, if the user gets impatient and hits ctrl-c, they can still benefit from the work we've done up to that point.

Merge request reports