Skip to content

tools: add more options to track flaky tests

Rodrigo Muino Tomonari requested to merge github/fork/aduh95/track-flaky-tests into main
  • For tests that are listed as flaky, try running them 100 times until we get a passing one. If it keeps failing after 100 attempts, it's not flaky, it's broken.
  • For test failures from tests that are not marked as flaky, try to run them 10 times to get an idea if the test is flaky or broken.

Those changes only apply to GHA, Jenkins config is not on this repo AFAIK, and local runs are not affected as it doesn't change the defaults.

My hope is with such changes, we would be lenient on marking tests as flaky because the CI would check that are indeed flaky (currently, it would simply ignore its result).

Refs: https://github.com/nodejs/node/pull/43929#issuecomment-1193104729 /cc @tniessen

Merge request reports