tokio thread panic with IO error when targeting wasm32-wasi

[fakeshadow]

Version
1.22

Platform
Linux pop-os 6.0.6-76060006-generic #202210290932~1667401208~22.04~d2df702 SMP PREEMPT_DYNAMIC Wed N x86_64 x86_64 x86_64 GNU/Linux (Also tested on other popular distros like arch the same issue exists.)

Description

When using wasmtime cli to run tokio runtime powered wasm applcation there is occasional case where wasmtime produce an I/O error cause tokio to panic with following message:

xitca-web-wasm: thread 'main' panicked at 'unexpected error when polling the I/O driver: Os { code: 29, kind: Uncategorized, message: "I/O error" }', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/io/mod.rs:180:23
xitca-web-wasm: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
xitca-web-wasm: Error: failed to run main module `./target/wasm32-wasi/release/xitca-web-wasm.wasm`
xitca-web-wasm: Caused by:
xitca-web-wasm:     0: failed to invoke command default
xitca-web-wasm:     1: error while executing at wasm backtrace:
xitca-web-wasm:            0: 0x44ba0 - <unknown>!rust_panic
xitca-web-wasm:            1: 0x264c8 - <unknown>!std::panicking::rust_panic_with_hook::hd6e0bbfcaf5f6e64
xitca-web-wasm:            2: 0x45f06 - <unknown>!std::panicking::begin_panic_handler::{{closure}}::hba16a922b240a170
xitca-web-wasm:            3: 0x45e3c - <unknown>!std::sys_common::backtrace::__rust_end_short_backtrace::hc38a923ae4d0f643
xitca-web-wasm:            4: 0x40351 - <unknown>!rust_begin_unwind
xitca-web-wasm:            5: 0x14a4 - <unknown>!core::panicking::panic_fmt::hfd6ef3977462d3a0
xitca-web-wasm:            6: 0x4ab53 - <unknown>!tokio::runtime::io::Driver::turn::h4db71c728c8dad60
xitca-web-wasm:            7: 0x49e74 - <unknown>!tokio::runtime::time::Driver::park_internal::h1fca15a0c7fbcad8
xitca-web-wasm:            8: 0x49019 - <unknown>!tokio::runtime::scheduler::current_thread::Context::park::ha377a4805dc54869
xitca-web-wasm:            9: 0x38eda - <unknown>!xitca_server::server::Server::new::hd8d6d90952a38ca0
xitca-web-wasm:           10: 0x346a2 - <unknown>!xitca_web_wasm::main::h4be9a8ba215e974b
xitca-web-wasm:           11: 0x251a8 - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::h0313bb192bbe64a3
xitca-web-wasm:           12: 0x251ce - <unknown>!std::rt::lang_start::{{closure}}::h59bba1161d685204
xitca-web-wasm:           13: 0x3d4c1 - <unknown>!__original_main
xitca-web-wasm:           14:  0x725 - <unknown>!_start
xitca-web-wasm:           15: 0x5d946 - <unknown>!_start.command_export
xitca-web-wasm:        note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable to may show more debugging information
xitca-web-wasm:     2: wasm trap: wasm `unreachable` instruction executed
---------------------------------------------------------

According to this line error kind with Uncategorized variant is excluded from error handling causing the panic:

tokio/tokio/src/runtime/io/mod.rs

Lines 175 to 179 in 28ec4a6

	#[cfg(tokio_wasi)]
	Err(e) if e.kind() == io::ErrorKind::InvalidInput => {
	// In case of wasm32_wasi this error happens, when trying to poll without subscriptions
	// just return from the park, as there would be nothing, which wakes us up.
	}

After further look into the source of error it leads to mio where event error is treated the same as selector error:
https://github.com/tokio-rs/mio/blob/0accf7dc22f197245e6a1aa84096262cd6f6e4d4/src/sys/wasi/mod.rs#L113

I did not look further into how wasmtime produce this error due to not familiar with it's code base but I can confirm it did not happen on windows 11 when I test the same code.

Reproducible example (docker needed):

git clone https://github.com/TechEmpower/FrameworkBenchmarks 
cd FrameworkBenchmarks
./tfb --test xitca-web-wasm

[rvolosatovs]

I can consistently trigger this panic server-side by ungracefully closing the client-side connection accepted on the listener side in WASI. It seems that it occurs on write to the (closed) stream
In 1.24.2 TCP listener is essentially unusable in WASI environment (since any client can kill the server remotely).
Is there any progress on this issue? Maybe some pointers to what could be a possible fix?

I will try to bisect when I have more capacity

[fakeshadow]

I have a simple fix that work for my use case. fakeshadow/mio@332d265

I'm not sure if it's the intended or correct way of handling event error hence no PR from it. Hope it can help you fixing it.

[Darksonn]

It sounds pretty weird. I'm not really sure what's going on in your change to mio.

Thoughts @Thomasdezeeuw ?

[Thomasdezeeuw]

According to this line error kind with Uncategorized variant is excluded from error handling causing the panic:

tokio/tokio/src/runtime/io/mod.rs

Lines 175 to 179 in 28ec4a6

	#[cfg(tokio_wasi)]
	Err(e) if e.kind() == io::ErrorKind::InvalidInput => {
	// In case of wasm32_wasi this error happens, when trying to poll without subscriptions
	// just return from the park, as there would be nothing, which wakes us up.
	}

This is unrelated to the panic, this line checks for InvalidInput, which is different from Uncategorized. Though I think perhaps we should handle this within Mio, not in Toko, but that's outside of the scope of this issue.

The actual panic comes from line 180:

tokio/tokio/src/runtime/io/mod.rs

Line 180 in 28ec4a6

Err(e) => panic!("unexpected error when polling the I/O driver: {:?}", e),

After further look into the source of error it leads to mio where event error is treated the same as selector error: https://github.com/tokio-rs/mio/blob/0accf7dc22f197245e6a1aa84096262cd6f6e4d4/src/sys/wasi/mod.rs#L113

This is how error handling is done for many events in wasm. If it wasn't then nothing wasm related would run.

I have a simple fix that work for my use case. fakeshadow/mio@332d265

I'm not sure if it's the intended or correct way of handling event error hence no PR from it. Hope it can help you fixing it.

This just ignored all errors.

The relevant error code is 29, which is an "I/O error" (a nice and detailed error message), see https://github.com/bytecodealliance/wasi/blob/984fa4fc17a074e8ea720cf319ad5e0357a991c7/src/lib_generated.rs#L129. Which the WASI spec doesn't say anything more about, see https://github.com/WebAssembly/WASI/blob/main/phases/snapshot/docs.md#errno the io variant.

So the next step would be to determine where this error comes from within wasmtime, but I don't have time to do that today.

[Darksonn]

I'm marking this issue as help-wanted. I have no experience with wasmtime, and it would be a big help if someone can investigate what the error means.