Skip to content

test: refactor tests for easier isolated execution

Rodrigo Muino Tomonari requested to merge github/fork/bcoe/test-refactor into master

Problem Statement

I've been eyeballing test coverage reports, as a way of finding subsystems where I can pitch in and help write more tests. In this process I believe I've identified a few aspects of the testing feedback loop that can be improved:

  1. running tests with coverage takes forever (there isn't an easy way to drill down to run coverage for just one submodule, e.g., http2.).
  2. it's confusing which folder new tests should be put in; some newer features place tests in a submodule-specific folder, e.g, @bmeck's tests for es-modules can be found in ./test/es-modules, other tests are placed in the catch-all ./test/parallel folder.
  3. there are currently thousands of tests in the ./test/parallel folder, which makes it hard to easily work with in some text editors.
  4. several variables need to be updated when new test folders are added, e.g., CI_JS_SUITES.

Proposed Solution

This pull requests introduces several mostly backwards-compatible changes to the test runner, in an effort to streamline the testing feedback loop:

  1. CI_NATIVE_SUITES and CI_JS_SUITES have been made configurable, so that make coverage can be run against a subset of suites, e.g., CI_JS_SUITES=child-process* CI_NATIVE_SUITES= make coverage.

  2. tools/test.py has been refactored, such that it now performs a recursive walk of folders and supports nested suites, making it easier to group tests by subsystem: screen shot 2017-09-16 at 1 01 58 am

  3. ./test/parallel can be split into subsystem specific suites, using the refactored tools/test.py.

  4. the special variable DEFAULT_JS_SUITES has been introduced. When this is observed test.py will use a programmatically generated list of reasonable-default test suites.

    it's worth noting, that in this process I removed known_issues from the default list of tests. I'm concerned that this would drive up coverage metrics for tests that we know are failing; make test-known-issues can be run periodically in its stead.

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • documentation is changed or added
  • commit message follows commit guidelines
Affected core subsystem(s)

test, build

CC: @addaleax, @MylesBorins, @jasnell

Merge request reports