Get devices used for a graph after graph compilation

[philloooo]

Hi! This is deriving from discussion here #815 (comment) .
Eventually which devices are used can only be known after a model is provided. It's a different problem from the context level device query mechanism so I decided to create a separate issue.

I've tested with CoreML's MLComputePlan which can give you op level device selections after a model is compiled. (Before calling dispatch).
Tflite also gives which delegate gets used for each op, after a graph is compiled.

So for WebNN, we can attach such information to MLGraph which represents a compiled graph.
The first high level information we can attach is a list of devices that will be used to execute the graph, since sometimes a graph could be executed using multiple devices.

const graph = await builder.build({'C': C});
console.log(graph.devices) // ['cpu', 'npu']

We could further expose op level device selection with a map or list:

const graph = await builder.build({'C': C});
console.log(graph.devices) // ['cpu', 'npu']

console.log(graph.deviceForOperations)

{
  "add_1": "cpu",
  "conv2d_2": "npu"
}

The only thing is we would need identifiers for each op, we could: 1. auto generate op identifiers using op name + auto incrementing index. 2. Use label and only return ops that have labels defined.

Maybe just the high level graph.devices is good enough for app developers to make decisions. But I want to provide what's possible with current backends right now.

[mwyrzykowski]

What is the use case for the website in providing the device information?

[zolkis]

Given the discussion in #815 (comment, comment), the developer use case (for frameworks, not for websites) seems to be the following (please correct/update/complete this):

• Before downloading/loading a model, the developer wants to know if e.g. the GPU can be used for inference with WebNN.
• If no, then they might want to try other path than WebNN.
• If yes, then in some cases (e.g. CoreML) the model needs to be dispatched before knowing for sure whether it can be executed on GPU. For that we'd need this API discussed here and implemented in define graph.devices #854 .
  Based on the answer, the developer can choose another option than WebNN.

Besides that, the feature permits gathering some data on typical graph accelerations (fingerprintable) which might help the spec work on device selection API.

[mwyrzykowski]

I'm not sure if this provides the website with any useful or actionable information. The model could run on one device at one point in time but the same exact model could run on a different device 100ms later for instance.

So if we did provide this information, we could return 'cpu' when the developer queries due to system conditions despite if the model was run repeatedly, it would end up on the gpu or npu. I.e., the underlying frameworks restrict us from providing any useful information to the framework or website.

It would be great to have a more concrete use case prior to adding this to the WebNN API.

[philloooo]

So if we did provide this information, we could return 'cpu' when the developer queries due to system conditions despite if the model was run repeatedly, it would end up on the gpu or npu. I.e., the underlying frameworks restrict us from providing any useful information to the framework or website.

Thanks for sharing that, I didn't know that it would return differently according to system resources. I think this can be addressed by making the API return async graph.devices() instead and describe in the spec that this returns the devices that would get used at the current system situation.

Re: concrete use case, we discussed a bit more and think it would be best to build a demo app that exercises the hypothetical use cases we have in mind and see if the graph.devices is necessary. We will report back on how that goes.

[mwyrzykowski]

Re: concrete use case, we discussed a bit more and think it would be best to build a demo app that exercises the hypothetical use cases we have in mind and see if the graph.devices is necessary. We will report back on how that goes.

Oh great, that would certainly be useful, thank you.