resource limit exceeded when calling go guest functions compiled by tinygo

[wujunzhuo]

Test Case

I want to make a performance test for Go guest functions, hence export an empty function hello:

package main

func main() {}

//export hello
func hello() {}

Compile: tinygo build -o hello.wasm -target wasi

Then I use wasmtime crate to run the function repeatedly:

use wasmtime::{Engine, Linker, Module, Store};
use wasmtime_wasi::WasiCtxBuilder;

fn main() {
    let engine = Engine::default();
    let wasi = WasiCtxBuilder::new().build();
    let mut store = Store::new(&engine, wasi);
    let module = Module::from_file(&engine, "./hello.wasm").unwrap();

    let mut linker = Linker::new(&engine);
    wasmtime_wasi::add_to_linker(&mut linker, |s| s).unwrap();
    linker.module(&mut store, "", &module).unwrap();
    let func = linker
        .get(&mut store, "", "hello")
        .unwrap()
        .into_func()
        .unwrap()
        .typed::<(), ()>(&store)
        .unwrap();

    loop {
        func.call(&mut store, ()).unwrap();
    }
}

Expected Results

The program should be continuously running without any error.

Actual Results

wujunzhuo@wujunzhuo-macbook demo % ./target/debug/hello
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: resource limit exceeded: instance count too high at 10001', src/bin/hello.rs:22:35
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Versions and Environment

Wasmtime version or commit: 5.0.0

Operating system: MacOS 13.1

Architecture: M1, MacBook Air 2020

[bjorn3]

I think tinygo produces a wasi command and not a wasi reactor. This means that loop { func.call(&mut store, ()).unwrap(); } will for every loop iteration create a new instance, initialize it, run the function and then destroys it again. Currently any state associated with a Store (including wasm instances) do not get destroyed until the entire store is destroyed. This means your loop would cause a memory leak. As precaution every Store by default has a ResourceLimiter set to allow at most 10000 instances, which is what you hit.

I think the following would work by using a separate store for every instantiation but I haven't tested it:

use wasmtime::{Engine, Linker, Module, Store};
use wasmtime_wasi::WasiCtxBuilder;

fn main() {
    let engine = Engine::default();
    let wasi = WasiCtxBuilder::new().build();
    let mut store = Store::new(&engine, wasi);
    let module = Module::from_file(&engine, "./hello.wasm").unwrap();

    let mut linker = Linker::new(&engine);
    wasmtime_wasi::add_to_linker(&mut linker, |s| s).unwrap();
    linker.module(&mut store, "", &module).unwrap();
    let func = linker
        .get(&mut store, "", "hello")
        .unwrap()
        .into_func()
        .unwrap()
        .typed::<(), ()>(&store)
        .unwrap();

    loop {
        let fresh_wasi = WasiCtxBuilder::new().build();
        let mut fresh_store = Store::new(&engine, fresh_wasi);
        func.call(&mut fresh_store, ()).unwrap();
    }
}

[dgryski]

Is there anything tinygo can do here to make it nicer to play with?

[bjorn3]

Wasi has two execution models:

• Command: This is roughly equivalent to an executable. Exactly one user specified function may be called (or _start if no explicit function is chosen). After that the wasm instance must be destroyed. A command can be identified by the presence of the _start function even if another function is chosen to be called instead.
• Reactor: This is roughly equivalent to a library. The _initialize function (if any) is executed before any other exports are called and then any number of different calls to exports may be performed. The wasm instance can stay alive as long as you want.

See https://github.com/WebAssembly/WASI/blob/069e3a4aee7ecaa8b52bd3c0ebe4fc055c1b1951/legacy/application-abi.md#current-unstable-abi for a more formal specification.

Clang uses -mexec-model to switch between the command and reactor model and rustc -Zwasi-exec-model. I think it would be helpful if tinygo got a flag to switch between the two too. Basically don't export _start for a reactor, but instead _initialize and from this _initialize only initialize the runtime environment of tinygo and not run the main function. By the way I don't know if you do already, but I believe you have to add code to every exported function to initialize the runtime environment when using the command model.

[wujunzhuo]

@bjorn3 👍 Your explanations about the wasi execution model spec is very clear~

However, it seems that we couldn't use another new store object since the linker is binding with the original one. If I initialize the whole wasi runtime and store in the loop, the memory leak problem could be overcome; but this will lead to significant performance degradation...

[bjorn3]

However, it seems that we couldn't use another new store object since the linker is binding with the original one.

I looked at the source and it seemed like it would for for wasi commands provided that you never use linker.instantiate or similar, but if you tried it, I guess I missed something.

If I initialize the whole wasi runtime and store in the loop, the memory leak problem could be overcome; but this will lead to significant performance degradation...

Yeah

[alexcrichton]

It's a known issue that command modules (like the one I think you're using here) are easy to leak memory with. Note though that you're not required to execute in a command-like fashion and use Linker::module. You could instead use Linker::instantiate which would not bind the linker to a store, meaning you could use the same Linker across many stores for each instantiation.

Note that Wasmtime has had a lot of effort put into making instantiation fast, and it's expected to be viable for applications to make a new instance per request, for example.

I'll also note that there's nothing TinyGo specific here, it's more about the embedder and how the module is run.

[wujunzhuo]

Note though that you're not required to execute in a command-like fashion and use Linker::module. You could instead use Linker::instantiate which would not bind the linker to a store, meaning you could use the same Linker across many stores for each instantiation.

@alexcrichton I changed the code as per your suggestion. This time the program could work normally without memory leak anymore~

use wasmtime::{Engine, Linker, Module, Store};
use wasmtime_wasi::WasiCtxBuilder;

fn main() {
    let engine = Engine::default();
    let wasi = WasiCtxBuilder::new().build();
    let mut store = Store::new(&engine, wasi);
    let module = Module::from_file(&engine, "./hello.wasm").unwrap();

    let mut linker = Linker::new(&engine);
    wasmtime_wasi::add_to_linker(&mut linker, |s| s).unwrap();
    let instance = linker.instantiate(&mut store, &module).unwrap();
    linker.instance(&mut store, "", instance).unwrap();
    let func = instance
        .get_typed_func::<(), ()>(&mut store, "hello")
        .unwrap();

    loop {
        func.call(&mut store, ()).unwrap();
    }
}

[alexcrichton]

Ok sounds good. I'm going to close this issue as this leak is probably best tracked by #1913 and I believe everything else is accounted for.

Note that in your latest example linker.instance(&mut store, "", instance).unwrap(); can be removed.