Skip to content

tools: fix skip detection of test runner output

Rodrigo Muino Tomonari requested to merge github/fork/richardlau/testharnessskip into main

Fix the Python test harness so that it no longer treats the # skipped part of the summary at the end of the built-in test runner output as marking the test as skipped.

Fixes: https://github.com/nodejs/node/issues/50346

So it turns out that the Python test runner scans the output of the tests to look for SKIP directives: https://github.com/nodejs/node/blob/d335487e3f39437a5e3cc5a4a07bf253b9d0d505/tools/test.py#L86 https://github.com/nodejs/node/blob/d335487e3f39437a5e3cc5a4a07bf253b9d0d505/tools/test.py#L388-L391 (this regex is based on http://testanything.org/tap-specification.html#skipping-tests).

For the tests that are using the built-in test runner, this is matching the # skipped line in the summary at the end of the output, e.g.

$ ./node --test --test-reporter=tap test/parallel/test-dotenv-node-options.js
TAP version 13
# Subtest: .env supports NODE_OPTIONS
    # Subtest: should have access to permission scope
    ok 1 - should have access to permission scope
      ---
      duration_ms: 26.0744
      ...
    # Subtest: validate only read permissions are enabled
    ok 2 - validate only read permissions are enabled
      ---
      duration_ms: 22.168635
      ...
    # Subtest: TZ environment variable
    ok 3 - TZ environment variable
      ---
      duration_ms: 19.970299
      ...
    1..3
ok 1 - .env supports NODE_OPTIONS
  ---
  duration_ms: 69.387679
  type: 'suite'
  ...
1..1
# tests 3
# suites 1
# pass 3
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 144.850498
$

which causes the Python test harness to insert a SKIP directive with the number of skipped tests (e.g. 0 in this case) as the reason for skipping:

$ ./tools/test.py --progress=tap parallel/test-dotenv-node-options
TAP version 13
1..1
ok 1 parallel/test-dotenv-node-options # skip 0
  ---
  duration_ms: 213.30700
  ...

All tests passed.
$

With the updated regex:

$ ./tools/test.py --progress=tap parallel/test-dotenv-node-options
TAP version 13
1..1
ok 1 parallel/test-dotenv-node-options
  ---
  duration_ms: 212.83200
  ...

All tests passed.
$

cc @nodejs/test_runner

Merge request reports