WASI testsuite

[wanderer]

It would be nice to have a standard testsuite for the WASI API

[mohanson]

Is there an official testsuit currently available? Even very simple.

[tschneidereit]

There's a testsuite for the wasi-sdk, which is perhaps the closest we have right now.

I'm not sure it'd make sense to do something more directly for WASI itself (as opposed to the libc abstraction on top of it, which wasi-sdk uses) right now: once WASI is defined in terms of Interface Types[1], I think the tests would probably have to look somewhat different.

[1]: see the last paragraph in the design principle doc's section on security.

[RReverser]

I'm not sure it'd make sense to do something more directly for WASI itself (as opposed to the libc abstraction on top of it, which wasi-sdk uses)

I didn't realise these are separate; but yeah, the testsuite of wasi-sdk looks pretty comprehensive. Do you think it's something that could be extracted into a separate submodule that other implementations could rely on / contribute to independently from wasi-sdk implementation?

[RReverser]

Maybe I rushed with "comprehensive" bit; looking through the suite, it doesn't seem to test most of filesystem operations, for example, or randomness, or timers.

https://github.com/devsnek/node-wasi/tree/master/test/c by @devsnek, referenced from one of related issues above, seems better in this regard.

[devsnek]

We've got more tests now: https://github.com/nodejs/node/tree/master/test/wasi/c

[mohanson]

This is exactly what I want to express, there are many wasm vm implementations in the world, and they all have their own wasi test cases, which seems inconsistent.

I think we can write a unified testsuit, these testcases are not so platform-dependent, and which can ensure that the data structure, function signature and function logic are correct.

Some existing test cases are too complicated, I mean "not so much like unit tests". For example, when I want to testing fd_readdir, the case imports too many other wasi functions like:

  (import "wasi_unstable" "proc_exit" (func $__wasi_proc_exit (type 2)))
  (import "wasi_unstable" "fd_readdir" (func $__wasi_fd_readdir (type 3)))
  (import "wasi_unstable" "fd_close" (func $__wasi_fd_close (type 4)))
  (import "wasi_unstable" "fd_fdstat_get" (func $__wasi_fd_fdstat_get (type 5)))
  (import "wasi_unstable" "path_open" (func $__wasi_path_open (type 6)))
  (import "wasi_unstable" "args_sizes_get" (func $__wasi_args_sizes_get (type 5)))
  (import "wasi_unstable" "args_get" (func $__wasi_args_get (type 5)))
  (import "wasi_unstable" "fd_prestat_get" (func $__wasi_fd_prestat_get (type 5)))
  (import "wasi_unstable" "fd_prestat_dir_name" (func $__wasi_fd_prestat_dir_name (type 0)))
  (import "wasi_unstable" "fd_seek" (func $__wasi_fd_seek (type 7)))
  (import "wasi_unstable" "fd_write" (func $__wasi_fd_write (type 8)))

[caspervonb]

I have some tests for Deno's std/wasi implementation here and there are some legacy tests for libc in the pre-merge repository I haven't brought over; overall fairly similar coverage to @devsnek's tests, looks like I go a bit overboard on explicitly testing stdio 🤔 .

Ultimately I'd like to have a bunch of tests in the WebAssembly text format (for sharing between toolchains) that test the snapshots directly rather than relying fully on integration tests.

[sunfishcode]

Tests written in wat are great because they easily work in many engines and don't require users to have various compilers for various languages installed in order to run them. So if people have tests of this form and/or are interesting in writing such tests, it'd be great to collect them.

That said though, tests written in wat directly, and targeting WASI APIs directly, are often very verbose and difficult to maintain, so it'd be good to collect compiled tests as well. Ideally we'd have the original Rust/C/etc. source files in the repository, so that people with the appropriate toolchains installed can edit and debug, and also the compiled output in the repository, in either wat or wasm format, so that it's possible for others to run the tests without having the toolchains installed.

[sbc100]

I agree with @sunfishcode .. I don't think we should require that tests be authored in wat. Writing them in C or rust and providing supplying the compiled output in wat/wasm format should be fine.

[RReverser]

Sounds like merging Node & Deno tests and precompiling to WAT alongside existing C files would be a good start then. @devsnek & @caspervonb do you want to uh talk to each other? :)

[binji]

That said though, tests written in wat directly, and targeting WASI APIs directly, are often very verbose and difficult to maintain, so it'd be good to collect compiled tests as well.

Another alternative could be to generate wat files from another language, JS/Python. We've done that for a few proposals now, see https://github.com/WebAssembly/bulk-memory-operations/tree/master/test/meta and https://github.com/WebAssembly/simd/tree/master/test/core/simd/meta as examples.