Support additional Execution Providers in ONNX `wasi-nn` backend

[kaivol]

Feature

Currently the ONNX backend in wasmtime-wasi-nn only uses the default CPU execution provider and ignores the ExecutionTarget requested by the WASM caller.

wasmtime/crates/wasi-nn/src/backend/onnxruntime.rs

Lines 21 to 33 in 24c1388

	fn load(&mut self, builders: &[&[u8]], target: ExecutionTarget) -> Result<Graph, BackendError> {
	if builders.len() != 1 {
	return Err(BackendError::InvalidNumberOfBuilders(1, builders.len()).into());
	}
	let session = Session::builder()?
	.with_optimization_level(GraphOptimizationLevel::Level3)?
	.with_model_from_memory(builders[0])?;
	let box_: Box<dyn BackendGraph> =
	Box::new(ONNXGraph(Arc::new(Mutex::new(session)), target));
	Ok(box_.into())
	}

I would like to suggest adding support for additional execution providers (CUDA, TensorRT, ROCm, ...) to wasmtime-wasi-nn.

Benefit

Improved performance for WASM modules using the wasi-nn API.

Implementation

ort already has support for many execution providers, so integrating these into wasmtime-wasi-nn should not be to much work.
I would be interested in looking into this, however, I only really have the means to test the DirectML and NVIDIA CUDA / TensorRT EPs.

Alternatives

Leave it to the users to add support for additional execution providers.

[abrown]

I was looking at old issues and ran across this one (sorry for such a late reply!): I completely agree with this idea. I am tempted to say "go for it!" but maybe there is some coordination needed. E.g., I think @jianjunz has started enabling some DirectML bits in #8756. And @devigned may have some opinions on the best way to do this. But from my perspective, this seems like a worthwhile avenue to pursue.

[devigned]

I think this is a great idea! One interesting part will be testing. We may need to spin up some hardware to make sure the functionality stays evergreen.

[zhen9910]

Is there any update for this open issue? We also want to use GPU/NPU for ONNX backend for our projects. If not available yet, I may try to take a look and want to get some suggestions.

[zhen9910]

I made some changes to support GPU for onnx backend, please review #12044

It updated the onnxruntime crate ort to 2.0.0-rc.10 with improved cuda support and added onnx-cuda in wasmtime-wasi-nn onnx backend. I verified it using an Azure-VM installed with Nvidia A10 GPU.

[abrown]

I'm going to close this in favor of #12044. Never mind, this is an issue resolved by #12044.