test: refactor tests for easier isolated execution
Problem Statement
I've been eyeballing test coverage reports, as a way of finding subsystems where I can pitch in and help write more tests. In this process I believe I've identified a few aspects of the testing feedback loop that can be improved:
- running tests with coverage takes forever (there isn't an easy way to drill down to run coverage for just one submodule, e.g., http2.).
- it's confusing which folder new tests should be put in; some newer features place
tests in a submodule-specific folder, e.g, @bmeck's tests for
es-modules
can be found in./test/es-modules
, other tests are placed in the catch-all./test/parallel
folder. - there are currently thousands of tests in the
./test/parallel
folder, which makes it hard to easily work with in some text editors. - several variables need to be updated when new test folders are added, e.g.,
CI_JS_SUITES
.
Proposed Solution
This pull requests introduces several mostly backwards-compatible changes to the test runner, in an effort to streamline the testing feedback loop:
-
CI_NATIVE_SUITES
andCI_JS_SUITES
have been made configurable, so thatmake coverage
can be run against a subset of suites, e.g.,CI_JS_SUITES=child-process* CI_NATIVE_SUITES= make coverage
. -
tools/test.py
has been refactored, such that it now performs a recursive walk of folders and supports nested suites, making it easier to group tests by subsystem: -
./test/parallel
can be split into subsystem specific suites, using the refactoredtools/test.py
. -
the special variable
DEFAULT_JS_SUITES
has been introduced. When this is observedtest.py
will use a programmatically generated list of reasonable-default test suites.it's worth noting, that in this process I removed
known_issues
from the default list of tests. I'm concerned that this would drive up coverage metrics for tests that we know are failing;make test-known-issues
can be run periodically in its stead.
Checklist
-
make -j4 test
(UNIX), orvcbuild test
(Windows) passes -
documentation is changed or added -
commit message follows commit guidelines
Affected core subsystem(s)
test, build
CC: @addaleax, @MylesBorins, @jasnell