Skip to content

test: automate WPT updates with drivers in test/wpt

Rodrigo Muino Tomonari requested to merge github/fork/joyeecheung/wpt-any into master

This is a big diff but note that the biggest commit (the first one) is automated by git node wpt. For a high-level design see test/wpt/README.md in this diff.

There is still work to do with the upstream to port more tests to .js files and split them properly so that we only need to skip the parts that truely cannot be run by us (e.g. ones that requires Response/Request objects).

Currently there are still a few test-whatwg-* tests in test/parallel that are copy-pasted with manually added fixtures/linter switches, but I think we can land these patches first to eliminate the manual maintenance of of the tests that can be migrated, and gradually migrate the rest of them later while working with the upstream.

Refs: https://github.com/nodejs/node/issues/23192

Current status:

  • console tests: all migrated here
  • URL tests:
    • Some of them are still in .html files upstream
    • Some of them mixes JS tests and <a>/<area> tests with references to document.createElement in .window.js, or needs something like Request/Response which we don't have
    • There are a bunch of test-whatwg-url-custom-* tests that can be upstreamed and pulled here later
    • Others are migrated here
  • encoding tests: I have a WIP branch for them - but will need to figure out the ICU requirements in the CI first as some of the subtests need more support than the others, and we will probably want to split them into separate files somehow in the upstream as well, so I'd prefer to leave them in a subsequent PR

Note that none of the IDL tests currently pass - I plan to fix them later after this PR lands.

test: use git node wpt to pull WPT into test/fixtures

This patch uses the git node wpt command in node-core-utils to initialize test/fixtures/wpt with a subset of the Web Platform Tests (https://github.com/web-platform-tests/wpt), including:

  • console tests
  • url tests
  • interfaces (.idl WebIDL files)
  • test harness located in resources, including the WebIDL parser

Also initializes the README.md, LICENSE and versions.json in the test/fixtures/wpt.

In later patches, we will use these files to run WPT and move away from the manually maintained copy-pasted files in test/parallel/test-whatwg-*.

https://github.com/nodejs/node-core-utils/blob/master/docs/git-node.md#git-node-wpt

test: remove WPT tests that are now .any.js in the upstream

In favor of actual .any.js tests that will be run with the WPT harness in a subsequent patch.

The other types of .js tests (e.g. .window.js tests) and the .html tests remained to be supported or migrated in the upstream.

test: use URL fixtures under test/fixtures/wpt/url/resources

Removes the following files:

  • test/fixtures/url-tests.js
  • test/fixtures/url-setter-tests.js
  • test/fixtures/url-toascii.js

in favor of:

  • test/fixtures/wpt/url/resources/urltestdata.json
  • test/fixtures/wpt/url/resources/setters_tests.json
  • test/fixtures/wpt/url/resources/toascii.json

Also removes dependency of fixtures/url-tests.js in http2 tests and use fixtures/person-large.jpg instead since they are just looking for a big enough file to transfer.

test: initialize test/wpt to run URL and console .js tests

This patch:

  • Creates a new test suite wpt that can be used to run a subset of Web Platform Tests
  • Adds a WPTRunner in test/common/wpt.js that can run the WPT subset in test/fixtures/wpt with a vm and the WPT harness while taking the status file in test/wpt/status into account. Here we use a new format of status file (in JSON) to handle specific requirements (like ICU requirements) in the tests and to handle expected failures and TODOs.
  • Adds documentation on how the runner and the update automation works
  • Runs the WHATWG URL tests and the console tests with the new test runner.

With this patch we eliminates the need of copy-pasting with manual modifications to update a large chunk of our WPT subset previously maintained in test/parallel. Now the tests run in test/wpt can be automatically updated with git node wpt without modifications by the actual WPT harness instead of our home-grown mock.

There are still a few URL tests left that need to be migrated in the upstream to be placed in .js instead of .html - we currently still use the legacy harness mock in the test files.

doc: remove legacy WPT integration guide

Point to the new guide in test/wpt/README.md instead.

Merge request reports