Interoperability between browsers is critical to Chromium's mission of improving the web. We believe that leveraging and contributing to a shared test suite is one of the most important tools in achieving interoperability between browsers. The web-platform-tests repository is the primary shared test suite where all browser engines are collaborating. There's also a csswg-test repository for CSS tests, but that will soon be merged into web-platform-tests.
[TOC]
Chromium has mirrors (web-platform-tests, csswg-test) of the GitHub repos, and periodically imports a subset of the tests so that they are run as part of the regular Blink layout test testing process.
The goal of this process are to be able to run the Web Platform Tests unmodified locally just as easily as we can run the Blink tests, and ensure that we are tracking tip-of-tree in the Web Platform Tests repository as closely as possible, and running as many of the tests as possible.
There is an automatic process for updating the Chromium copy of the Web Platform Tests. The import is done by the builder w3c-test-autoroller builder.
The easiest way to check the status of recent imports is to look at:
- Recent logs on Buildbot for w3c-test-autoroller builder
- Recent CLs created by blink-w3c-test-autoroller@chromium.org.
We control which tests are imported via a file called W3CImportExpectations, which has a list of directories to skip while importing.
In addition to the directories and tests explicitly skipped there, tests may also be skipped for a couple other reasons, e.g. if the file path is too long for Windows. To check what files are skipped in import, check the recent logs for w3c-test-autoroller builder.
To pull the latest versions of the tests that are currently being imported, you can also directly invoke the wpt-import script.
That script will pull the latest version of the tests from our mirrors of the upstream repositories. If any new versions of tests are found, they will be committed locally to your local repository. You may then upload the changes.
If you wish to add more tests (by un-skipping some of the directories currently
skipped in W3CImportExpectations), you can modify that file locally and commit
it, and on the next auto-import, the new tests should be imported. If you want
to import immediately, you can also run wpt-import --allow-local-commits.
If you need to make changes to Web Platform Tests, just commit your changes directly to LayoutTests/external/wpt and the changes will be automatically upstreamed within 24 hours.
Note that tests in Web Platform Tests are expected to match behavior defined by the relevant WHATWG or W3C specification, not simply Blink's behavior. If in doubt, please request code review from someone with expertise in the relevant specification text.
Note: if you're adding a new test in external/wpt, you'll need to re-generate
MANIFEST.json manually until CL 2644783003 is
landed. The command to do so is:
Tools/Scripts/webkitpy/thirdparty/wpt/wpt/manifest --work \
--tests-root=LayoutTests/external/wptThe email you commit with in Chromium will be the author of the commit on GitHub. You can add it as a secondary address on your GitHub account to link your exported commits to your GitHub profile.
This cannot be avoided entirely as the two repositories are independent, but should be rare with frequent imports and exports. When it does happen, manual intervention will be needed and in non-trivial cases you may be asked to help resolve the conflict.
It's still possible to make direct pull requests to web-platform-tests. The processes for getting new tests committed the W3C repos are at http://testthewebforward.org/docs/. Some specifics are at http://testthewebforward.org/docs/github-101.html.