A shared testing multi-staging framework

[bnjbvr]

TL;DR: http://bnjbvr.github.io/wasm-spec/test/out/ (expect failures; harness is a bit broken at the moment)

Some rough explanation there.

Not ready for merge yet:

• the harness is a bit broken (because of top level calls but also other issues I need to investigate first). Tests not passing in Chromium are mainly the harness failing in a dumb way; in Firefox we've diverged a bit from the spec, so many more failures are expected (a failed compilation or instanciation may lead to a big amount of failures).
• not sure where to merge it: ideally, it'd be on the gh-pages branch so people can use the landing page there. But the master and gh-pages branches have diverged long time ago on this repository, and my gh-pages branch on my fork is based off this master. Opinions?

[rossberg]

The way the gh-pages branch is used by GH is very weird -- it requires a structure that is pretty much incompatible with how anybody would want to organise master. So no sources should live there, I think, you only push stuff over (in custom ways) for publishing. At least that's the recommended workflow as I understood it from some tutorials.

On the other hand, the converted JS files should not be checked into master, they only need to exist on the published page. So I think we should have some script for performing the build+publish step, part of which would be (re)generating the converted files.

[jfbastien]

Can the live pages be part of webassembly.org instead of some github page? @s3ththompson

I'd rather minimize the number of web pages we have. We can squirrel this two levels deep to avoid confusion.

[s3ththompson]

Yes. We can pull in the source into the webassembly/website the same way we do for the design docs.

If others are on board with this, can you open an issue on the website repo?

[bnjbvr]

Can the live pages be part of webassembly.org instead of some github page?

If we do so, then each new test added by a PR to this repository (be it a wast, JS or HTML) test will need another build / PR to the webassembly.org repository. This sounds like something we'd like to avoid (because of the manual sync need and scattering), ideally, but @rossberg-chromium previous comment about using the master and gh-pages seems to imply the same (modulo we'd need to have 1 PR for each branch, instead of 1 PR per repo).

It sound to me the only way to have a single consistent repository would be having a completely separate repository test, using only on a gh-pages branch (thus browsable at webassembly.org/test, iiuc), but I have heard the intent was to remove repositories, not add any new ones.

[jfbastien]

I'd like webassembly.org to be automated anyways.

[bnjbvr]

For the sake of simplicity, and after some struggle to respect the order of execution (involving a queue of Promise factories), I've reverted back to using a simple synchronous test harness for the wast harness, since we can now (thanks to the functional run and call abstractions).

To wit: if a module can't be instantiated (because of an impl quirk, for instance), and this module is used in further tests, we have the choice of either not running the further tests using what should be a module, or failing them too. I think the latter is simpler, less error-prone and more explicit, even thought it inflates the number of failures by large factors. Any disagreements?

[bnjbvr]

To not block other people from working on these tests, I propose to review and merge this one. Subsequent work will include being able to pass misc command-line options to the build.py script (to build only the front page for webassembly.org, or to build the JS tests or the HTML tests for browser CI, or a combination of these).

@rossberg-chromium @mtrofin, PTAL.

[mtrofin]

lgtm

[rossberg]

Nit: V8

[rossberg]

Nit: I'd prefer not to front-load the bullet list with too much information. Maybe we should have sections on each one below.

[rossberg]

Din't look at all the details, but LGTM so far. The only nit I have is the name lib for the infrastructure directory, since test/lib sounds too much like "tests of the library" next to the other dirs. How about util or something else?

[rossberg]

Or, in fact, why not simply call the lib dir test/harness?

[bnjbvr]

Or, in fact, why not simply call the lib dir test/harness?

lib/ is often used as a placeholder name in web development for putting utilities, but harness is more descriptive, so I'll take this. Thanks!

[bnjbvr]

Din't look at all the details,

If that can make reviewing easier: from all the files in this PR,

• testharness.js and testharness.css have been been taken from the WPT repository. testharnessreport.js contains parameters for testharness.js and is very small and contained.
• wasm-constants and wasm-module-builder have been taken from V8's repository.
• build.py is likely to change in future PRs to allow building only for the shell, for the WPT, or the front page (since building everything all the time doesn't quite make sense).

[rossberg]

Thanks, lgtm

[bnjbvr]

This is ready for merge, whenever everybody is agreeing and comfortable doing so.