Code Coverage in wasm-bindgen-test?

[anp]

Motivation

I'd like to be able to use -Zprofile when building my wasm-bindgen tests with wasm-pack and have those tests produce coverage files. I currently run my tests in a headless browser with wasm-pack and an ideal solution would have a clear integration story there with a good out of the box experience.

Proposed Solution

The instrumentation emitted by default writes coverage profiling data to disk in a location relative to the binary being run. If there's a nice way to override those callbacks for a single binary/target then wasm-bindgen could presumably collect the coverage buffers in its JS test harness and forward them to the CLI for persisting to disk.

Alternatives

I'm still looking at LLVM's coverage docs and I'm not sure if it's straightforward to override the instrumentation collection buffers like that. If it's not then I'd be willing to compile in a larger (wasi?) runtime into my tests to provide the needed filesystem I/O. I think the LLVM instrumentation at least offers environment variables to control the paths where data is written.

Additional Context

I looked for related issues and couldn't find any. I think that "test artifact extraction" from wasm tests might also be useful for supporting snapshot testing but I'm not familiar enough with relevant internals to say for sure.

[alexcrichton]

To some degree WebAssembly instrumentation and coverage should be easier in theory than native binaries because it's easy to change/inject instructions into a binary (e.g. at the top/end of all blocks). That would require work investing into some sort of wasm transform, however, which would require a side table (like dwarf info) about what instruction corresponds to what.

Otherwise LLVM's infrastructure may work with tweaks, but AFAIK no work has been done towards that. I would suspect you'd probably want to start out with wasm32-wasi since it'll be easier as many things already "just work", but if something works there seems reasonable to try to extend it to wasm-bindgen!

[anp]

Makes sense and thanks for the advice! I am skimming https://motley-coder.com/2019/03/31/webassembly-coverage/ and it looks like compiler-rt support for the target is the main thing.

[Amanieu]

You could try using minicov which embeds a stripped down version of the LLVM profiling runtime. It has no dependencies on libc or any external code so it should compile on WASM. You'll need to write out the profraw file yourself though (see the README).

[bbarwik]

Hey. I've created a tutorial how to generate code coverage for webassembly projects. This one is dedicated for blockchains using WASM VM, however I think it may be useful to you - https://github.com/hknio/code-coverage-for-webassembly

[fzyzcjy]

Hi, is there any updates? Thanks!

[aDogCalledSpot]

I looked into this a bit and tried to simply compile a test using minicov.

wasm_bindgen_test::wasm_bindgen_test_configure!(run_in_browser);

#[cfg(test)]
mod test {
    #[wasm_bindgen_test::wasm_bindgen_test]
    async fn test_foo() {
        assert!(true);
    }
}

then run RUSTFLAGS="-Cprofile-generate -Zno-profiler-runtime" wasm-pack test --firefox --headless.

Output:

[2024-01-09T12:58:06Z DEBUG wasm_bindgen_wasm_interpreter] starting a call of Id { idx: 12653 } Some("__wbindgen_describe___wbg_window_57e9bd3b31ab8737.command_export")
[2024-01-09T12:58:06Z DEBUG wasm_bindgen_wasm_interpreter] arguments []
[2024-01-09T12:58:06Z DEBUG wasm_bindgen_wasm_interpreter] starting a call of Id { idx: 3133 } Some("__wasm_call_ctors")
[2024-01-09T12:58:06Z DEBUG wasm_bindgen_wasm_interpreter] arguments []
[2024-01-09T12:58:06Z DEBUG wasm_bindgen_wasm_interpreter] starting a call of Id { idx: 3138 } Some("__llvm_profile_init")
[2024-01-09T12:58:06Z DEBUG wasm_bindgen_wasm_interpreter] arguments []
[2024-01-09T12:58:06Z DEBUG wasm_bindgen_wasm_interpreter] starting a call of Id { idx: 3137 } Some("__llvm_profile_register_functions")
[2024-01-09T12:58:06Z DEBUG wasm_bindgen_wasm_interpreter] arguments []
[2024-01-09T12:58:06Z DEBUG wasm_bindgen_wasm_interpreter] starting a call of Id { idx: 0 } Some("__llvm_profile_register_function")
[2024-01-09T12:58:06Z DEBUG wasm_bindgen_wasm_interpreter] arguments [1205344]
thread 'main' panicked at crates/wasm-interpreter/src/lib.rs:222:18:
can only call locally defined functions

The last function to be called __llvm_profile_register_function is the first function from the shown stack that is part of minicov.

So it seems as if the issue is that we would have to be able to call functions through FFI but the wasm-interpreter doesn't support this since it's really only supposed to be used for much more simple things.

We need a way to skip these functions since they are irrelevant to the unit under test. First approaches are having complications with keeping the stack properly in order.

If we can skip through these functions and make it to the point where the generated wasm is actually mounted by the node/chrome/firefox, then we can see how to retrieve the coverage from minicov.

[aDogCalledSpot]

I believe that the issue is that we are trying to measure coverage of the describe function created by #[wasm_bindgen]. The profiler doesn't know nor care that this macro expanded code so it just gets treated the same way as every other code. A simple solution could be to add #[coverage(off)] to describe but it's an unstable feature so you would also have to add #![feature(coverage_attribute)] to the appropriate path and that's what I'm currently struggling with.

[aDogCalledSpot]

I compiled rustc locally and removed the feature gate of #[coverage(off)]. Unfortunately, I wasn't able to get the desired results. For now, what works is simply ignoring any i64 related instructions and saying that these only exist because of coverage.

I can now at least run my code with coverage profiles. Will now look into getting the coverage information back out of the tests.