test: automate WPT updates with drivers in test/wpt
This is a big diff but note that the biggest commit (the first one) is automated by git node wpt
. For a high-level design see test/wpt/README.md
in this diff.
There is still work to do with the upstream to port more tests to .js files and split them properly so that we only need to skip the parts that truely cannot be run by us (e.g. ones that requires Response/Request objects).
Currently there are still a few test-whatwg-*
tests in test/parallel
that are copy-pasted with manually added fixtures/linter switches, but I think we can land these patches first to eliminate the manual maintenance of of the tests that can be migrated, and gradually migrate the rest of them later while working with the upstream.
Refs: https://github.com/nodejs/node/issues/23192
Current status:
- console tests: all migrated here
- URL tests:
- Some of them are still in
.html
files upstream - Some of them mixes JS tests and
<a>
/<area>
tests with references todocument.createElement
in.window.js
, or needs something likeRequest
/Response
which we don't have - There are a bunch of
test-whatwg-url-custom-*
tests that can be upstreamed and pulled here later - Others are migrated here
- Some of them are still in
- encoding tests: I have a WIP branch for them - but will need to figure out the ICU requirements in the CI first as some of the subtests need more support than the others, and we will probably want to split them into separate files somehow in the upstream as well, so I'd prefer to leave them in a subsequent PR
Note that none of the IDL tests currently pass - I plan to fix them later after this PR lands.
test: use git node wpt to pull WPT into test/fixtures
This patch uses the git node wpt
command in node-core-utils
to initialize test/fixtures/wpt
with a subset of the Web
Platform Tests (https://github.com/web-platform-tests/wpt),
including:
- console tests
- url tests
- interfaces (.idl WebIDL files)
- test harness located in
resources
, including the WebIDL parser
Also initializes the README.md, LICENSE and versions.json in the
test/fixtures/wpt
.
In later patches, we will use these files to run WPT
and move away from the manually maintained copy-pasted files
in test/parallel/test-whatwg-*
.
https://github.com/nodejs/node-core-utils/blob/master/docs/git-node.md#git-node-wpt
test: remove WPT tests that are now .any.js in the upstream
In favor of actual .any.js tests that will be run with the WPT harness in a subsequent patch.
The other types of .js tests (e.g. .window.js tests) and the .html tests remained to be supported or migrated in the upstream.
test: use URL fixtures under test/fixtures/wpt/url/resources
Removes the following files:
- test/fixtures/url-tests.js
- test/fixtures/url-setter-tests.js
- test/fixtures/url-toascii.js
in favor of:
- test/fixtures/wpt/url/resources/urltestdata.json
- test/fixtures/wpt/url/resources/setters_tests.json
- test/fixtures/wpt/url/resources/toascii.json
Also removes dependency of fixtures/url-tests.js
in http2 tests
and use fixtures/person-large.jpg
instead since they are just
looking for a big enough file to transfer.
test: initialize test/wpt to run URL and console .js tests
This patch:
- Creates a new test suite
wpt
that can be used to run a subset of Web Platform Tests - Adds a
WPTRunner
intest/common/wpt.js
that can run the WPT subset intest/fixtures/wpt
with a vm and the WPT harness while taking the status file intest/wpt/status
into account. Here we use a new format of status file (in JSON) to handle specific requirements (like ICU requirements) in the tests and to handle expected failures and TODOs. - Adds documentation on how the runner and the update automation works
- Runs the WHATWG URL tests and the console tests with the new test runner.
With this patch we eliminates the need of copy-pasting with manual
modifications to update a large chunk of our WPT subset previously
maintained in test/parallel
. Now the tests run in test/wpt
can
be automatically updated with git node wpt
without modifications
by the actual WPT harness instead of our home-grown mock.
There are still a few URL tests left that need to be migrated in the upstream to be placed in .js instead of .html - we currently still use the legacy harness mock in the test files.
doc: remove legacy WPT integration guide
Point to the new guide in test/wpt/README.md instead.