Skip to content

Commit

Permalink
Update WPT documentation and add note on wpt_internal
Browse files Browse the repository at this point in the history 
Bug: 852053
Change-Id: I9b3d78fe60361fe1156eb02b396c401584ff15ba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2289399
Commit-Queue: Stephen McGruer <smcgruer@chromium.org>
Reviewed-by: Luke Z <lpz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#787228}
  • Loading branch information
@stephenmcgruer
stephenmcgruer authored and Commit Bot committed Jul 10, 2020
1 parent 710c6a5 commit a12b34f
Showing 1 changed file with 175 additions and 155 deletions.
330 changes: 175 additions & 155 deletions docs/testing/web_platform_tests.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,137 @@ web-platform-tests, including tips for writing and reviewing tests.

[TOC]

## Writing tests

To contribute changes to web-platform-tests, just commit your changes directly
to [web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt)
and the changes will be automatically upstreamed within 24 hours.

Changes involving adding, removing or modifying tests can all be upstreamed.
Any changes outside of
[external/wpt](../../third_party/blink/web_tests/external/wpt) will not be
upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`,
will also not be upstreamed.

Running the web tests will automatically regenerate MANIFEST.json to pick up
any local modifications.

Most tests are written using testharness.js, see
[Writing Web Tests](./writing_web_tests.md) and
[Web Tests Tips](./web_tests_tips.md) for general guidelines.

### Write tests against specifications

Tests in web-platform-tests are expected to match behavior defined by the
relevant specification. In other words, all assertions that a test makes
should be derived from a specification's normative requirements, and not go
beyond them. It is often necessary to change the specification to clarify what
is and isn't required.

When implementation experience is needed to inform the specification work,
[tentative tests](https://web-platform-tests.org/writing-tests/file-names.html)
can be appropriate. It should be apparent in context why the test is tentative
and what needs to be resolved to make it non-tentative.

### Tests that require testing APIs

[testdriver.js](https://web-platform-tests.org/writing-tests/testdriver.html)
provides a means to automate tests that cannot be written purely using web
platform APIs, similar to `internals.*` and `eventSender.*` in regular Blink
web tests.

If no testdriver.js API exists, check if it's a
[known issue](https://github.com/web-platform-tests/wpt/labels/testdriver.js)
and otherwise consider filing a new issue. For instructions on how to add a new
testing API, see [WPT Test Automation for
Chromium](https://docs.google.com/document/d/18BpD41vyX1cFZ77CE0a_DJYlGpdvyLlx3pwXVRxUzvI/preview#)

An alternative is to write manual tests that are automated with scripts from
[wpt_automation](../../third_party/blink/web_tests/external/wpt_automation).
Injection of JS in manual tests is determined by `loadAutomationScript` in
[testharnessreport.js](../../third_party/blink/web_tests/resources/testharnessreport.js).

Such tests still require case-by-case automation to run for other browser
engines, but are more valuable than purely manual tests.

Manual tests that have no automation are still imported, but skipped in
[NeverFixTests](../../third_party/blink/web_tests/NeverFixTests); see
[issue 738489](https://crbug.com/738489).

### Adding new top-level directories

Entirely new top-level directories should generally be added upstream, since
that's the only way to add an OWNERS file upstream. After adding a new top-level
directory upstream, you should add a line for it in `W3CImportExpectations`.

Adding the new directory (and `W3CImportExpectations` entry) in Chromium and
later adding an OWNERS file upstream also works.

### Will the exported commits be linked to my GitHub profile?

The email you commit with in Chromium will be the author of the commit on
GitHub. You can [add it as a secondary address on your GitHub
account](https://help.github.com/articles/adding-an-email-address-to-your-github-account/)
to link your exported commits to your GitHub profile.

If you are a Googler, you can also register your GitHub account at go/github,
making it easier for other Googlers to find you.

### What if there are conflicts?

This cannot be avoided entirely as the two repositories are independent, but
should be rare with frequent imports and exports. When it does happen, manual
intervention will be needed and in non-trivial cases you may be asked to help
resolve the conflict.

### Direct pull requests

It's still possible to make direct pull requests to web-platform-tests, see
https://web-platform-tests.org/writing-tests/github-intro.html.

### wpt_internal

It is sometimes desirable to write WPT tests that either test Chromium-specific
behaviors, or that cannot yet be upstreamed to WPT (e.g. because the spec is
very nascent). For these cases, we maintain a separate directory,
[wpt_internal](../third_party/blink/web_tests/wpt_internal) that runs under the
WPT testing infrastructure (e.g. uses wptserve, etc), but which is not
upstreamed to WPT.

Please see the `wpt_internal`
[README](../third_party/blink/web_tests/wpt_internal/README) for more details.

**Note**: A significant downside of `wpt_internal` is that your tests may be
broken by upstream changes to the resources scripts (e.g. `testharness.js`), as
`wpt_internal` does not use the forked version of `testharness.js` used by all
other non-`external/wpt` tests. Use of [WPT-NOTIFY](#wpt_notify) is recommended
to ensure you are notified of breakages.

## Running tests

Same as Blink web tests, you can use
[`run_web_tests.py`](web_tests.md#running-the-tests) to run any WPT test.

One thing to note is that glob patterns for WPT tests are not yet supported.

See [Running WPT tests in Content Shell](web_tests_in_content_shell.md#Running-WPT-Tests-in-Content-Shell)
for debugging etc.

## Reviewing tests

Anyone who can review code and tests in Chromium can also review changes in
[external/wpt](../../third_party/blink/web_tests/external/wpt)
that will be automatically upstreamed. There will be no additional review in
web-platform-tests as part of the export process.

If upstream reviewers have feedback on the changes, discuss on the pull request
created during export, and if necessary work on a new pull request to iterate
until everyone is satisfied.

When reviewing tests, check that they match the relevant specification, which
may not fully match the implementation. See also
[Write tests against specifications](#Write-tests-against-specifications).

## Importing tests

Chromium has a [mirror](https://chromium.googlesource.com/external/w3c/web-platform-tests/)
Expand Down Expand Up @@ -74,14 +205,38 @@ output.
Note that we are considering making WPT-NOTIFY opt-out instead of opt-in: see
https://crbug.com/845232

### Enabling import for a new directory

If you wish to add more tests (by un-skipping some of the directories currently
skipped in `W3CImportExpectations`), you can modify that file locally and commit
it, and on the next auto-import, the new tests should be imported.

If you want to import immediately (in order to try the tests out locally, etc)
you can also run `wpt-import`, but this is not required.

Remember your import might fail due to GitHub's limit for unauthenticated
requests, so consider [passing your GitHub credentials](#GitHub-credentials) to
the script.

### Skipped tests

We control which tests are imported via a file called
[W3CImportExpectations](../../third_party/blink/web_tests/W3CImportExpectations),
which has a list of directories to skip while importing.

In addition to the directories and tests explicitly skipped there, tests may
also be skipped for a couple other reasons, e.g. if the file path is too long
for Windows. To check what files are skipped in import, check the recent logs
for [wpt-importer builder][wpt-importer].

### Waterfall failures caused by automatic imports.

If there are new test failures that start after an auto-import,
there are several possible causes, including:

1. New baselines for flaky tests were added (http://crbug.com/701234).
2. Modified tests should have new results for non-Release builds but they weren't added (http://crbug.com/725160).
3. New baselines were added for tests with non-deterministic test results (http://crbug.com/705125).
1. New baselines for flaky tests were added (https://crbug.com/701234).
2. Modified tests should have new results for non-Release builds but they weren't added (https://crbug.com/725160).
3. New baselines were added for tests with non-deterministic test results (https://crbug.com/705125).

Because these tests are imported from the Web Platform tests, it is better
to have them in the repository (and marked failing) than not, so prefer to
Expand All @@ -91,7 +246,7 @@ can fix it manually.

[wpt-importer]: https://ci.chromium.org/p/infra/builders/luci.infra.cron/wpt-importer

### Automatic export process
## Exporting tests

If you upload a CL with any changes in
[third_party/blink/web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt),
Expand Down Expand Up @@ -126,6 +281,22 @@ For maintainers:

[wpt-exporter]: https://ci.chromium.org/p/infra/builders/luci.infra.cron/wpt-exporter

## Notes for WPT infra maintainers

### Manual import

To pull the latest versions of the tests that are currently being imported, you
can also directly invoke the
[wpt-import](../../third_party/blink/tools/wpt_import.py) script.

That script will pull the latest version of the tests from our mirrors of the
upstream repositories. If any new versions of tests are found, they will be
committed locally to your local repository. You may then upload the changes.

Remember your import might fail due to GitHub's limit for unauthenticated
requests, so consider [passing your GitHub credentials](#GitHub-credentials) to
the script.

### GitHub credentials

When manually running the `wpt-import` and `wpt-export` scripts, several
Expand All @@ -144,154 +315,3 @@ unauthenticated requests, so it is recommended that you let `wpt-export` and
and `GH_TOKEN`, the access token you have just generated. After that,
pass `--credentials-json <path-to-json>` to `wpt-export` and
`wpt-import`.

### Manual import

To pull the latest versions of the tests that are currently being imported, you
can also directly invoke the
[wpt-import](../../third_party/blink/tools/wpt_import.py) script.

That script will pull the latest version of the tests from our mirrors of the
upstream repositories. If any new versions of tests are found, they will be
committed locally to your local repository. You may then upload the changes.

Remember your import might fail due to GitHub's limit for unauthenticated
requests, so consider [passing your GitHub credentials](#GitHub-credentials) to
the script.

### Skipped tests

We control which tests are imported via a file called
[W3CImportExpectations](../../third_party/blink/web_tests/W3CImportExpectations),
which has a list of directories to skip while importing.

In addition to the directories and tests explicitly skipped there, tests may
also be skipped for a couple other reasons, e.g. if the file path is too long
for Windows. To check what files are skipped in import, check the recent logs
for [wpt-importer builder][wpt-importer].

### Enabling import for a new directory

If you wish to add more tests (by un-skipping some of the directories currently
skipped in `W3CImportExpectations`), you can modify that file locally and commit
it, and on the next auto-import, the new tests should be imported.

If you want to import immediately (in order to try the tests out locally, etc)
you can also run `wpt-import`, but this is not required.

Remember your import might fail due to GitHub's limit for unauthenticated
requests, so consider [passing your GitHub credentials](#GitHub-credentials) to
the script.

## Writing tests

To contribute changes to web-platform-tests, just commit your changes directly
to [web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt)
and the changes will be automatically upstreamed within 24 hours.

Changes involving adding, removing or modifying tests can all be upstreamed.
Any changes outside of
[external/wpt](../../third_party/blink/web_tests/external/wpt) will not be
upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`,
will also not be upstreamed.

Running the web tests will automatically regenerate MANIFEST.json to pick up
any local modifications.

Most tests are written using testharness.js, see
[Writing Web Tests](./writing_web_tests.md) and
[Web Tests Tips](./web_tests_tips.md) for general guidelines.

### Write tests against specifications

Tests in web-platform-tests are expected to match behavior defined by the
relevant specification. In other words, all assertions that a test makes
should be derived from a specification's normative requirements, and not go
beyond them. It is often necessary to change the specification to clarify what
is and isn't required.

When implementation experience is needed to inform the specification work,
[tentative tests](https://web-platform-tests.org/writing-tests/file-names.html)
can be appropriate. It should be apparent in context why the test is tentative
and what needs to be resolved to make it non-tentative.

### Tests that require testing APIs

[testdriver.js](https://web-platform-tests.org/writing-tests/testdriver.html)
provides a means to automate tests that cannot be written purely using web
platform APIs, similar to `internals.*` and `eventSender.*` in regular Blink
web tests.

If no testdriver.js API exists, check if it's a
[known issue](https://github.com/web-platform-tests/wpt/labels/testdriver.js)
and otherwise consider filing a new issue. For instructions on how to add a new
testing API, see [WPT Test Automation for
Chromium](https://docs.google.com/document/d/18BpD41vyX1cFZ77CE0a_DJYlGpdvyLlx3pwXVRxUzvI/preview#)

An alternative is to write manual tests that are automated with scripts from
[wpt_automation](../../third_party/blink/web_tests/external/wpt_automation).
Injection of JS in manual tests is determined by `loadAutomationScript` in
[testharnessreport.js](../../third_party/blink/web_tests/resources/testharnessreport.js).

Such tests still require case-by-case automation to run for other browser
engines, but are more valuable than purely manual tests.

Manual tests that have no automation are still imported, but skipped in
[NeverFixTests](../../third_party/blink/web_tests/NeverFixTests); see
[issue 738489](https://crbug.com/738489).

### Adding new top-level directories

Entirely new top-level directories should generally be added upstream, since
that's the only way to add an OWNERS file upstream. After adding a new top-level
directory upstream, you should add a line for it in `W3CImportExpectations`.

Adding the new directory (and `W3CImportExpectations` entry) in Chromium and
later adding an OWNERS file upstream also works.

### Will the exported commits be linked to my GitHub profile?

The email you commit with in Chromium will be the author of the commit on
GitHub. You can [add it as a secondary address on your GitHub
account](https://help.github.com/articles/adding-an-email-address-to-your-github-account/)
to link your exported commits to your GitHub profile.

If you are a Googler, you can also register your GitHub account at go/github,
making it easier for other Googlers to find you.

### What if there are conflicts?

This cannot be avoided entirely as the two repositories are independent, but
should be rare with frequent imports and exports. When it does happen, manual
intervention will be needed and in non-trivial cases you may be asked to help
resolve the conflict.

### Direct pull requests

It's still possible to make direct pull requests to web-platform-tests, see
https://web-platform-tests.org/appendix/github-intro.html.

## Running tests

Same as Blink web tests, you can use
[`run_web_tests.py`](web_tests.md#running-the-tests) to run any WPT test.

One thing to note is that glob patterns for WPT tests are not yet supported.

See [Running WPT tests in Content Shell](web_tests_in_content_shell.md#Running-WPT-Tests-in-Content-Shell)
for debugging etc.

## Reviewing tests

Anyone who can review code and tests in Chromium can also review changes in
[external/wpt](../../third_party/blink/web_tests/external/wpt)
that will be automatically upstreamed. There will be no additional review in
web-platform-tests as part of the export process.

If upstream reviewers have feedback on the changes, discuss on the pull request
created during export, and if necessary work on a new pull request to iterate
until everyone is satisfied.

When reviewing tests, check that they match the relevant specification, which
may not fully match the implementation. See also
[Write tests against specifications](#Write-tests-against-specifications).

0 comments on commit a12b34f

Please sign in to comment.