WASI tests

This repository contains tests for WebAssembly System Interface (WASI) and a test executor for running WASI tests on a selected WebAssembly runtime.

WASI is a modular collection of standardized APIs. Currently, WASI has not reached a v1 with a defined set of APIs. However, a snapshot of experimental APIs exists (wasi_snapshot_preview1). The repository only holds tests of APIs included in this snapshot. It does not include tests for other in-progress proposals or other experimental APIs.

The test executor included in the repository can however be used to run tests defined for proposals along with tests defined in this repository.

Getting started

1. Clone repository Use prod/testsuite-base branch as it already includes precompiled test binaries (see Branch structure paragraph).

git clone --branch prod/testsuite-base git@github.com:WebAssembly/wasi-testsuite.git

2. Make sure there's already an adapter for the runtime in the adapters directory; if not, create one (see the doc for details).
3. Install python3
   1. Ubuntu

   $ sudo apt install python3 python3-pip
   

4. Install test runner dependencies:

python3 -m pip install -r test-runner/requirements.txt

5. Execute test suites from this repository

python3 test-runner/wasi_test_runner.py                                                  \
    -t ./tests/assemblyscript/testsuite/ `# path to folders containing .wasm test files` \
       ./tests/c/testsuite/                                                              \
       ./tests/rust/testsuite/                                                           \
    -r adapters/wasmtime.py # path to a runtime adapter

Optionally you can specify test cases to skip with the --exclude-filter option.

python3 test-runner/wasi_test_runner.py                                                  \
    -t ./tests/assemblyscript/testsuite/ `# path to folders containing .wasm test files` \
       ./tests/c/testsuite/                                                              \
       ./tests/rust/testsuite/                                                           \
    --exclude-filter examples/skip.json                                                  \
    -r adapters/wasmtime.py # path to a runtime adapter

The default executable in the adapter used for test execution can be overridden using TEST_RUNTIME_EXE variable. This only works with adapters defined in adapters/, and might not work with 3rd party adapters.

TEST_RUNTIME_EXE="wasmtime --wasm-features all" python3 test-runner/wasi_test_runner.py                                                  \
    -t ./tests/assemblyscript/testsuite/ \
    -r adapters/wasmtime.py

Contributing

All contributions are very welcome. Contributors can help with:

• adding or updating test cases,
• improving test execution and reporting,
• integrating with new WASM runtimes,

and many others. We appreciate both code contributions as well as suggestions and bug reports.

Developer guide

Here is some additional information for developers who are willing to contribute.

Directory structure

• test-runner - test executor scripts.
• tests - source code of WASI tests and build scripts. The folder contains subfolders for all supported languages.
• .github - CI workflow definitions.
• doc - additional documentation.

Cleaning up temporary resources

Some of the tests (e.g. pwrite-with-access) generate output artifacts and their existence can affect consecutive test executions. Tests should clean up the artifacts they generate, but there might be cases where the test fails early. Test runner will automatically delete all the files and directories in the test suite directory with the .cleanup suffix.

Programming languages for tests

The repository currently consists of tests implemented in the following languages:

• C (with wasi-libc)
• AssemblyScript
• Rust

The list of supported languages can be extended if needed.

Test

Branch structure

Apart from development branches for various features, we identify the following branches as critical (i.e. they won't be removed or force-updated):

• main - main branch of the repository. Use this branch for development (e.g. updating test cases, modifying test runner)
• prod/testsuite-base - the branch is an up-to-date fork of the main branch but it also includes precompiled binaries. Use this branch for simply running tests and validating WASM runtimes (see doc for details).
• prod/daily-test-results - the branch contains daily test results for supported WASM runtimes (at the moment, we only execute tests on wasmtime and WAMR). In the future we intend to publish those results to the website to provide users with additional input for selecting the runtime.