Concerning the structure of the Wasm spec testsuite

[Robbepop]

I am trying to implement Wasm memory64 support in Wasmi. However, I am a bit confused about where to find the correct test cases in the Wasm spec testsuite for it. Wasmi does not (yet) support all of Wasm 3.0, and not even all of Wasm 2.0. So far I have identified the following test cases, however, they are spread around the entire directory structure and not isolated anywhere as it is usually the case with proposals:

• proposals/custom-page-sizes/memory_max_i64
• proposals/memory64/float_exprs
• proposals/wasm-3.0/address64
• proposals/wasm-3.0/align64
• proposals/wasm-3.0/endianess64
• proposals/wasm-3.0/float_memory64
• proposals/wasm-3.0/memory_grow64
• proposals/wasm-3.0/memory_trap64
• proposals/wasm-3.0/memory64
• proposals/wasm-3.0/table_copy_mixed

In this list I am still missing table64, store64 etc. Also some .wast files seem to have distinct memory32 and memory64 tests in different files and other files merge the two. This is kinda inconsistent. What is the general structure of this?

[Robbepop]

Searching for memory i64 in VS Code in spectest folder resulted in the following files:
(Note sure this really uncovers everything and funnily it does not show up a single file under proposals/memory64.)

While this helps, I think the current situation is not very confusing. At least not for runtimes that do not support all the Wasm proposal until Wasm 3.0. It would be great if the Wasm spec testsuite had a better structure to be a decent solution for all the Wasm runtimes, not just the most up-to-date ones. Otherwise it is getting harder and harder for new Wasm runtimes to start compete with existing ones.

[sbc100]

The memory64 has been merged in the wasm-3.0 branch so that is where you will find all tests for memory64. We should remove memory64 from the list of ongoing proposals.

Also, the wasm-3.0 branch is IIUC due to be merged into the main branch so I think the wasm-3.0 directory will be gone soon too. (@dschuff @rossberg I thought maybe the merge had already happened but I guess not?)

For now, wasm-3.0 is where to find the memory64 tests, and they are combined with all the other wasm-3.0 proposals so I think you would need to support at least some of the other wasm-3.0 proposals to get memory64 tests running.

The test-suite repo tracks individual proposal while they are ongoing but I doesn't continue to track them separately once they are have been merged (IIUC).

[rossberg]

There are a couple of remaining editorial TODOs that I have to resolve before I can merge the 3.0 branch into main. Hoping to get to that soon.

@sbc100, would it make sense to move at least the tests specific to this proposal to their own subdirectory?

(That said, it won't work to organise the test suite entirely by proposals, since there generally are various interactions and intersections between different features and proposals.)

[sbc100]

There are a couple of remaining editorial TODOs that I have to resolve before I can merge the 3.0 branch into main. Hoping to get to that soon.

@sbc100, would it make sense to move at least the tests specific to this proposal to their own subdirectory?

That's not currently how the testsuite repo is setup, or how the automation currently work. It also seems of marginal value since it would only work well for proposals that add completely new test files, but not at all for proposals that only modify existing tests.

Another problem is that even the test files that are specific to memory64 already include references to other wasm-3.0 proposals such as multi-memory.

(That said, it won't work to organise the test suite entirely by proposals, since there generally are various interactions and intersections between different features and proposals.)

[Robbepop]

Yeah inter-dependency of test cases is a big problem in the grand scheme of the Wasm spec testsuite for Wasm runtimes that do not (yet) support all of the proposals such as Wasmi. I hope this won't mean that writing a new Wasm runtime in the future is getting even harder and harder. It should be in the interest of a specification such as Wasm to have a great variety of implementers and future implementations.

A file-tree structure is not great for test cases that depend on multiple Wasm proposals.
A solution is to tag individual test cases (read: .wast asserts for example) to indicate which Wasm proposals they require. Maybe this could also be inferred and a Wasm runtime that does not support the inferred Wasm proposals simply skips the test. However, other tests might then build upon the skipped test but won't use its proposals and this is where problems arise. For such inter-dependent .wast assertions we could introduce groups of assertions that are guarded by the Wasm proposals they depend on?

From the perspective of a Wasm runtime implementer this is certainly very important to keep in mind. Otherwise, we are going to lose valuable implementers over time; or lose out on potential ones.

[Robbepop]

The thing is the following: The Wasm spec will grow over time which is natural for such a specification. And at some point having to implement everything at once is going to be simply infeasible. Over time, this would kill off any attempts at new Wasm implementations and we are kinda forever stuck with what we already got.

My stance is that a great specification will always try to encourage new implementations to pop up. However, as it is now, the Wasm specification does not really care about any implementations that are not up-to-date which is hurting potential new implementations to take shape.

In my honest opinion every major successful specification was successful because it remained feasible to implement over time.

And it certainly is possible to change this and make Wasm future proof in that regard.

[rossberg]

We have discussed some form of test annotations before, specifically in the context of (negative) tests that conflict with new features. Technically, I think that wouldn't be particularly hard, e.g., with special comment tags. But it requires quite a bit of thought, work and infrastructure to pull off. If somebody volunteered to do that work and maintain it, I don't think anybody would be opposed, though. ;)

Of course, building new implementations of Wasm will inevitably be getting harder and harder regardless...

[Robbepop]

If somebody volunteered to do that work and maintain it, I don't think anybody would be opposed, though. ;)

That's great but also it means that so far nobody stepped up. Or were there already people interested in that? Where could I connect to some? Indeed changing all the existing tests is a lot of work but I think it could actually be automated in CI to check if a test fails the annotation.

Of course, building new implementations of Wasm will inevitably be getting harder and harder regardless...

Yes, but allowing the process of building a new Wasm implementation to be incremental would certainly help a lot.

[rossberg]

There weren't any specific people. I made the vague suggestion (probably on some issue discussion that I can't find it right now) that such a thing could solve the transition issue with negative tests. But that problem was too small for motivating such a substantial infrastructure and tooling effort.

[sbc100]

In my honest opinion every major successful specification was successful because it remained feasible to implement over time.

Maybe we can learn from other specification here? Do you know of other specs that do this? Does JS or HTML, for example? How? e.g. can I implement a JS engine without, say, arrow functions? Is there some mechanism that prevents arrow functions from creeping into the rest of the test suite over time?

[Robbepop]

But that problem was too small for motivating such a substantial infrastructure and tooling effort.

I wonder if the effort is actually so big in our case. We'd mostly only have to change the .wast format and tooling to parse .wast.

If we come up with a design that allows existing engines to simply ignore whatever changes we introduce while also allowing other engines to know which tests they can afford to run and which not we'd already be in a good position.

Maybe we can learn from other specification here? Do you know of other specs that do this? Does JS or HTML, for example? How? e.g. can I implement a JS engine without, say, arrow functions? Is there some mechanism that prevents arrow functions from creeping into the rest of the test suite over time?

I personally don't know of any specs. I don't think that JS is a good candidate to take inspiration from and RISC-V with its modular design also does not fit Wasm's non-modular nature and use cases. I think we are in a decent position to come up with our own solutions at this point.

We do not even need to flag single isolated test cases because it is not difficult to detect their used Wasm feature set and simply skip them in a Wasm implementation. What we do need is a way to group test cases together that depend on previous test runs and only run them together or not at all. This does not even need to be tied to the concept of Wasm proposals. Instead we'd already have a solution if we were able to group wast directives.

Then if a Wasm implementation runs the Wast directives and stumbles upon such a group it can run the directives one by one and as soon as one of them fails due to unsupported Wasm features it can ignore the rest. This would require minimal support from the Wasm implementations to detect Wasm features (which should be simple, I hope?) but won't put the burden of Wasm features on the Wasm spec itself.

Such a design could look as follows:

(group
    (module ..)
    (invoke ..)
    (assert_unlikable ..)
    ;; etc..
)

And a Wast parser would provide Wast directives as follows:

1. WastGroupStart
2. WastModule
3. WastInvoke
4. WastAssertUnlikable
5. etc ..
6. WastGroupEnd

Existing up-to-date Wasm implementations could simply ignore WastGroupStart and WastGroupEnd and new ones could use this information to their advantage. This design would even allow groups to be included in groups by using counters on the Wasm implementation side. Though this is probably not even needed.

[rossberg]

FWIW, the .wast format already has a way of grouping, namely through nested scripts. But few or no other tools implement those.

One problem is that there already is a lot of tooling around the .wast format, and this would affect all of it.

[titzer]

It would generally be helpful if at the very least new spec tests introduced by a proposal are in new files, so that existing spec test files are stable.

AFAICT most engines are working around the organization of spec tests by having their own additional suites, either in .wast or using internal mechanisms.