Support flexible input sizes

@fdwr

There are models that may require flexible input sizes that are determined at inference time.

Examples include some vision models that may need to work on multiple resolutions, e.g., MODNet (a model for real-time portrait matting), the model accepts input in shape of [batch_size,3,height,width].

Transformers may work on arbitrary input lengths. When using KV cache, they may also need to increase the KV cache length by 1 per each inference.

For speech recognition model, e.g., Whisper, the encoder needs to work on arbitrary input lengths, its input shape is [batch_size, feature_size, encoder_sequence_length]. The decoder (with past) needs to increase the KV cache length by 1 per each inference. i.e. the shape of input named "past_key_values.0.decoder.key" is [batch_size,6,past_decoder_sequence_length,64], the shape of corresponding output named "present.0.decoder.key" is [batch_size,6,past_decoder_sequence_length + 1,64].

For other language models, e.g., Qwen2.5-0.5B-Instruct, the model takes input in shape [batch_size,sequence_length]. Similarly, it also increases KV cache length at inference time, i.e. the shape of input "past_key_values.0.key" is [batch_size,2,past_sequence_length,64], the corresponding output "present.0.key" shape is [batch_size,2,past_sequence_length + 1,64]

Lack of the support for flexible input sizes increases the complexity of using WebNN for those models. For example, apps need to modify the model and fix the input size when compiling the WebNN graph. At the inference time, apps need to resize the image input, or pad the language input to maximum length before passing the input to WebNN graph.

Flexible input sizes are already supported by native frameworks. For example ONNX allows model inputs having dynamic dimension. At inference time, ONNX Runtime allows apps to specify arbitrary values for those dynamic dimensions. CoreML also supports flexible input shape and allows to set multiple size options or size ranges.

/cc @fdwr @xenova

A few considerations:

API role - what is the role of WebNN in the stack, and how high-level should it be? There's an analog in graphics API's for retained mode vs immediate mode, more like SVG/WPF vs OpenGL/D3D. WebNN is an API rather than a model file format (where a model file naturally needs these dynamic dimensions to be reused with different input shapes), and so it's been the onus of the caller to resolve these questions before it even reaches WebNN. It's not really a limitation of WebNN itself, but rather in the intermediate layers that translate from higher level representations into the WebNN API. You can imagine a simple layer between a TF/ONNX model that stamps out the explicit shapes, creates a few static WebNN graphs of different sizes, ideally sharing the constants between all instances. So from the API point-of-view, nothing is really missing, but that said, there are other considerations to weigh...
API impact - a number of operators will be impacted, including input (as the tensor MLOperandDescriptor.shape is potentially incomplete, not just a list of integers), shape (because currently any constructed operator can tell you it's shape, which may now need to return an optional list or list of optionals rather than a concrete list of integers), reshape (because you might need unspecified free dimensions or need operations like flatten/squeeze/unsqueeze), pooling with output size rounding, and possibly others like convTranspose.
Execution speed of such models - will it improve speed or hinder speed (vs the fixed input size approach offline)? Lazily rebuilding the graph multiple times on demand (either externally or internally) for changing inputs will likely cost more than executing multiple static iterations, but if you could cache multiple instantiations, it could be as-fast/faster.
Memory sharing - memory sharing of constants isn't really made easier/harder with dynamic shapes because one can always create the constants via the constant(MLTensor tensor) overload so they are shared by the context across multiple built graphs.
Latency of stages - if the input shapes are not known until dispatch, then suddenly dispatch becomes heavyweight, as memory planning would potentially need to be redone each time inputs change sizes, but dispatch really should be a light go-do-thing-at-light-speed anymore but rather than redoing shape inference and memory planning too.
Graph construction overhead - how much runtime is spent in just the graph construction phase (the JS calls to create the object)? Allowing a single graph (with dynamic shapes) to be bound to different input shapes to instantiate different instances might save JS call time.
Internal complexity - If multiple input shapes are passed to dispatch over time, is the previous graph invalidated, or do we keep the last few shapes in case we see the same shape again (lest we execute the whole graph loop again in the LLM)? Shouldn't this be user controlled?

For the above reasons (I'd like dispatch to still be light and for the user to have control over which instantiated shapes to preserve), then if we add dynamic shapes, I'd like an intermediate step WebNN between build and dispatch where full shape computation and memory planning can occur:

build graph (may be dynamic or static)
⭐ produce finalized graph with known bound input shapes
dispatch graph of static shapes

e.g.

...
const context = await navigator.ml.createContext();
const builder = new MLGraphBuilder(context);
const constant = builder.constant(descriptor, new Float32Array(4).fill(0.2));
...
const potentiallyDynamicShapeGraph = await builder.build({'C': C});

const [inputTensorA, inputTensorB, outputTensorC] = await ...;
context.writeTensor(inputTensorA, new Float32Array(4).fill(1.0));
...

const inputs = {'A': inputTensorA, 'B': inputTensorB};
const outputs = {'C': outputTensorC};

// Finalize shapes                                        ⭐
const knownShapeGraph = await potentiallyDynamicShapeGraph.computeShapes(inputs, outputs);
// knownShapeGraph is just an MLGraph where all inputs/outputs are resolved concretely.
// If all input shapes were already known, then the original MLGraph is already such a graph that can be
// passed directly to dispatch, and calling computeShapes (or some other name) just returns
// the same graph back. Any knownShapeGraph can be held onto by the caller for later iterations,
// and multiple known-shape-graphs of differing input shapes can be held for reuse.

context.dispatch(knownShapeGraph, inputs, outputs);
// Dispatch requires the MLGraph to have well defined shapes on its outputs, otherwise it's an error.

@fdwr

@fdwr

API role - ... and so it's been the onus of the caller to resolve these questions before it even reaches WebNN.

ONNX Runtime may pass dynamic shape tensors to its execution providers. WebNN EP currently has to fall operators with dynamic inputs back to Wasm EP.

Also /cc @guschmue

WebNN EP currently has to fall operators with dynamic inputs back to Wasm EP.

Note the DML EP executes models that support dynamic input shapes (not as efficiently as if you pass the expected shapes up front, but it still runs) by creating the operator on demand or delaying construction of the graph until shape information is known, and also by setting tensor affinity for shapes to CPU-side to avoid any GPU/NPU readback (see here and here and here).

To add some more context to the request, there are three different types of dynamic shapes usually used in neural network models:

Completely Unknown Sizes: The dimensions of a tensor are entirely undefined before inference.
Symbolic Sizes: Multiple tensor shape dimensions use the same symbol, which is assigned a concrete value before an inference run begins.
Tensor-Derived Sizes: A tensor's dimensions are determined by data from another tensor. These dimensions can change during an inference run or be based on the output of a previous run.

More information about these types can be found in the TensorRT Documentation.

Should WebNN support all three types of dynamic shapes?

Also, dynamic shapes can be either bound or unbound. A bound dynamic shape has a specified upper limit (and sometimes a lower limit) for its dimensions.

Since system memory is finite and reallocating memory during each inference run is inefficient, I'd highly recommended that WebNN supports only bound dynamic shapes. This approach allows the backend to fail gracefully during the validation and compilation stages if it lacks sufficient resources or if the underlying hardware cannot handle the specified upper-bound dimensions.

Related resolution from Kobe F2F:

RESOLUTION: study more backends and do prototyping before more formally specifying solution

Add another computer vision use case requiring flexible input sizes support

Use Case: Paddle OCR for text extraction from images.

Models:

https://huggingface.co/webnn/PP-OCRv4-ONNX/tree/main
- ch_PP-OCRv4_det.onnx: [p2o.DynamicDimension.0,3,p2o.DynamicDimension.1,p2o.DynamicDimension.2]
- ch_PP-OCRv4_rec.onnx: [p2o.DynamicDimension.0,3,48,p2o.DynamicDimension.1]
https://huggingface.co/webnn/PP-OCRv5-ONNX/tree/main
- ch_PP-OCRv5_det.onnx: [DynamicDimension.0,3,DynamicDimension.1,DynamicDimension.2]
- ch_PP-OCRv5_rec.onnx: [DynamicDimension.0,3,48,DynamicDimension.1]

Testing:

Test URL: Paddle OCR Test
Test Source Code: GitHub Repository

This use case highlights the need for flexible input sizes to accommodate dynamic dimensions in OCR models, enabling effective text extraction from images.

To try out the WebNN spec, I built a tool (see https://github.com/rustnn/webnn-graph/ ) that converts ONNX dynamic graphs into WebNN static graphs, and then back to ONNX for execution via the ML graph builder. This round trip is meant to validate that the WebNN spec is sufficiently expressive.

This approach worked well for:

MobileNet - https://github.com/rustnn/rustnn/blob/main/examples/mobilenetv2_from_hub.py
all-MiniLM-L6-v2 - https://github.com/rustnn/rustnn/blob/main/examples/minilm_embeddings.py

However, it does not work for text generation models that rely on dynamic shapes. I spent a significant amount of time trying to eliminate the dynamic parts and make it work. In theory this seemed feasible, but in practice it does not appear possible for modern LLMs.

The core issue is constant folding. It works fine for simpler models, but for LLMs it bakes in fixed input sizes, breaks variable-length inputs, and prevents a growing KV cache. You end up needing a pre-allocated cache, which is suboptimal. And the graph works for a fixed token size. huh.

As a result, text generation models are effectively unsupported, or at least I do not see a viable way to support them with the current constraints.

I was curious how ONNX manages to support text generation with the WebNN backend, but it appears that this is not a pure WebNN backend, according to: #883 (comment)

Since dynamic shapes are a major use case, +1 for adding support for dynamic shapes.

@tarekziade

@tarekziade

I was curious how ONNX manages to support text generation with the WebNN backend, but it appears that this is not a pure WebNN backend, according to: #883 (comment)

Operators with dynamic input size would fall back to Wasm EP, that causes suboptimal performance.

As I shared at TPAC, developers typically need to make the following changes of text generation models to work with static input sizes and avoid fallbacks:

Prepare separate static models for the Prefill and Decoding stages. Each model is compiled with fixed input shapes.
The sequence length (sequence_length) is set to a constant value, typically max_sequence_length. Also applies to input_ids, position_ids, and attention_mask.

Cons of this approach:

Higher memory usage: one more static model compiled
Double complication time
Inefficiency for variable-length inputs in Prefill phase: All inputs must be padded to max_sequence_length
Increase deployment complexity: Managing multiple static models

/cc @Honry

right, thanks for the explanation. This is roughly what I was thinking about when I said fixed token size.

What if we added in MLOperandDescriptor.shape some optional symbolic dims, we would bind at execution time in compute() ? and some kind of updateSlice for KV cache write at decoding time?

@tarekziade

@tarekziade

What if we added in MLOperandDescriptor.shape some optional symbolic dims, we would bind at execution time in compute() ?

I prototyped MLDynamicDimension that supports specifying name (symbolic dim) and max size (@mtavenrath 's recommendation on bound dynamic shapes). Dynamic dimension can be used for creating input operand. The size validation would rely on max size if it is a dynamic dimension. The concreate dimension value is specified when calling the dispatch(). dispatch() will validate that the dynamic dimension with same name have same concrete value.

The major IDL change looks like

+dictionary MLDynamicDimension {
+  required DOMString name;
+  required [EnforceRange] unsigned long maxSize;
+};

+typedef ([EnforceRange] unsigned long or MLDynamicDimension) MLDimension;

+dictionary MLInputOperandDescriptor {
+  required MLOperandDataType dataType;
+  required sequence<MLDimension> shape;
+};

interface MLGraphBuilder {
- MLOperand input(USVString name, MLOperandDescriptor desc);
+ MLOperand input(USVString name, MLInputOperandDescriptor desc);

interface MLOperand {
   readonly attribute MLOperandDataType dataType;
-  readonly attribute FrozenArray<unsigned long> shape;
+  readonly attribute FrozenArray<MLDimension> shape;
 };

The Chromium prototype is available at https://chromium-review.googlesource.com/c/chromium/src/+/7496954

I've verified it with two samples:

Image classification MobileNet in webnn-samples

The input can be dynamic now

this.builder_.input('input',
      {dataType: 'float32',
       shape: [{name: 'batch', maxSize: 3},
               3,
               {name: 'height', maxSize: 512},
               {name: 'width', maxSize: 512}]});

The code is available at: huningxin/webnn-samples@b928643

Text embedding all-MiniLM-L6-v2 model (based on @tarekziade 's webnn-graph)

The input can be dynamic:

  const sequenceLengthDim = {name: "sequenceLength", maxSize: 128};
  env.set("attention_mask", builder.input("attention_mask", { dataType: "int64", shape: [1, sequenceLengthDim] }));
  env.set("input_ids", builder.input("input_ids", { dataType: "int64", shape: [1, sequenceLengthDim] }));
  env.set("token_type_ids", builder.input("token_type_ids", { dataType: "int64", shape: [1, sequenceLengthDim] }));

The app can change the sequenceLength according to the input sentence length at inference. The code is available at: huningxin/webnn-graph@273d36c

This model also uses a dynamic slice operator (the starts and ends are operand) to slice a constant operand (embeddings_position_ids) according to sequence length. I hesitated to introduce the dynamic slice operator because it would make WebNN operand shape is totally unknown which is hard to validate at build time. Instead, I changed it to an input which has dynamic shape, so user code can do the slice out of the model and feed to the dynamic input that runtime.

I am verifying bigger models (TinyLlama and Qwen etc.,), I'll share the results once they are available.

@miaobin

With great help from @miaobin, we verified that the latest Chromium prototype is able to run TinyLlama 1.1B model with dynamic input size. The sample code is available: building dynamic input size graph and inference code.

When building the model, user code can define dynamic dimensions and use them to create input operands, e.g. (I use batch size 1 for sake of simplicity).

const sequence_length = {name: 'sequence_length', maxSize: 2048};
const total_sequence_length = {name: 'total_sequence_length', maxSize: 2048};

const attention_mask = builder.input("attention_mask_593", { dataType: "int64", shape: [1, total_sequence_length] });
const input_ids = builder.input("input_ids_580", { dataType: "int64", shape: [1, sequence_length] });
const position_ids = builder.input("position_ids_622", { dataType: "int64", shape: [1, sequence_length] });

When doing prefill inference, user code can create input tensors according to the prompt input length, e.g.

const inputIdsTensor = await createWritableTensor('int64', [1, promptLen], 
    new BigInt64Array(prefillInputIds.map(t => BigInt(t))));
const attentionMaskTensor = await createWritableTensor('int64', [1, promptLen], 
    new BigInt64Array(prefillAttentionMask.map(t => BigInt(t))));
const positionIdsTensor = await createWritableTensor('int64', [1, promptLen], 
    new BigInt64Array(prefillPositionIds.map(t => BigInt(t))));

// Prepare KV cache tensors and dispatch graph
// ...

Then when running decode loop, user code can create input/position ids tensor of sequence length 1 and increase attention mask tensor's sequence length by 1 for each iteration, e.g.

let currentTotalLen = promptLen + 1;
const decodeInputIdsTensor = await createWritableTensor('int64', [1, 1], new BigInt64Array([0n]));
const decodePositionIdsTensor = await createWritableTensor('int64', [1, 1], new BigInt64Array([0n]));
// Decode loop: continue until maxSeqLen or EOS
for (let step = 1; !stopGeneration; step++) {
  // Update input tensors
  context.writeTensor(decodeInputIdsTensor, new BigInt64Array([BigInt(nextToken)]));
  context.writeTensor(decodePositionIdsTensor, new BigInt64Array([BigInt(currentTotalLen - 1)]));
                    
  // attention_mask shape is [1, total_sequence_length]
  // All values are 1 (indicating valid positions)
  const decodeAttentionMask = new BigInt64Array(currentTotalLen).fill(1n);
  const decodeAttentionMaskTensor = await createWritableTensor('int64', [1, currentTotalLen], decodeAttentionMask);

  // Prepare KV cache tensors and dispatch graph
  // ...
 
  currentTotalLen++;
}

By using one model for both prefill and decode stages, the session creation time and memory footprint are reduced significantly.

The prototype also introduces other two IDL changes for reshape and expand ops:

- MLOperand expand(MLOperand input, sequence<unsigned long> newShape, optional MLOperatorOptions options = {});
+ MLOperand expand(MLOperand input, sequence<MLDimension> newShape, optional MLOperatorOptions options = {});

- MLOperand reshape(MLOperand input, sequence<unsigned long> newShape, optional MLOperatorOptions options = {});
+ MLOperand reshape(MLOperand input, sequence<MLDimension> newShape, optional MLOperatorOptions options = {});

Both allow to use "known" dynamic dimension in the new shape, e.g. the TinyLlama model can reshape a tensor of shape [1, sequence_length, 32] to [1, sequence_length, 1, 1, 32] by builder.reshape(env.get("_2962"), [1,sequence_length,1,1,32])). And the model can also expand a tensor of shape [1, sequence_length] to shape [past_sequence_length, sequence_length] by builder.expand(env.get("_611"), [past_sequence_length,sequence_length]).

"Known" dynamic dimensions are dynamic dimensions that are defined by builder.input() method in a graph building session.

RESOLUTION: Create a spec PR for MLDynamicDimension to support models that may require flexible input sizes, supports both static and dynamic use cases.

@Honry

@Honry and @miaobin helped explore the SOTA open-source image generation model, Z-Image-Turbo. Z-Image-Turbo adopts a Scalable Single-Stream Diffusion Transformer (S3-DiT) that processes text and image tokens in a single unified attention stream. In particular, the image transformer model has two dynamic inputs: hidden_states of shape [1,16,1,height,width] which contains the noisy image latent, and encoder_hidden_states of shape [1,seq_len,2560] which contains the caption text embeddings. The image tokens and text tokens are padded to chucks (chunk size 32), concatenated and fed into transformers.

For example, the padding size is dynamic and calculated in PyTorch source code as:

# calculate padding size
cap_padding_len = (-cap_ori_len) % SEQ_MULTI_OF   # pad caption to multiple of 32

In the exported ONNX model, this "pad-to-chunks" algorithm is represented as:

Calculate the padding size: Shape → Gather → Mod → Sub → Mod — computes (-seq_len) % 32 = padding needed
Pad-to-chunks: Pad — pads [seq_len, 2560] → [seq_len + padding, 2560] (now a multiple of 32)

In Z-Image-Turbo, there are other usages depending on dynamic dimension calculation result. Those cases cannot be supported by only having input dynamic dimension in previous proposal. Bin and Wanming prototyped the following new operators and enabled Z-Image-Turbo successfully:

+ MLOperand mod(MLOperand a, MLOperand b, optional MLOperatorOptions options = {});

+ MLOperand shape(MLOperand input, optional MLOperatorOptions options = {});

+ MLOperand range(MLOperand start, MLOperand limit, MLOperand delta, optional MLOperatorOptions options = {});

+ MLOperand dynamicReshape(MLOperand input, MLOperand newShape, optional MLOperatorOptions options = {});

+ MLOperand dynamicExpand(MLOperand input, MLOperand newShape, optional MLOperatorOptions options = {});

+ MLOperand dynamicSlice(MLOperand input, MLOperand starts, MLOperand ends, optional MLOperatorOptions options = {});

+ MLOperand dynamicPad(MLOperand input, MLOperand pads, optional MLOperatorOptions options = {});

+ sequence<MLOperand> dynamicSplit(MLOperand input, MLOperand splits, unsigned long numOutputs, optional MLOperatorOptions options = {});

+ MLOperand dynamicResample2d(MLOperand input, MLOperand sizes, optional MLDynamicResample2dOptions options = {});

For those "dynamic" operators, the output operand sizes are unknown at the build-time and cannot be validated against limits. As the next step, we plan to explore the dispatch-time shape validation when all tensors' sizes are specified or can be inferred.

Bin and Wanming prototyped the following new operators and enabled Z-Image-Turbo successfully:

modulus is worth adding to WebNN as a core op primitive (on the 2025-Sep-23 list), but there's definite ambiguity regarding the sign (whether to follow the dividend or divisor) and whether to truncate or floor. Both operators, f(a, b) = a - (b * trunc(a / b)), and f(a, b) = a - (b * floor(a / b)), are useful. Rather than add an odd fmod attribute like ONNX did, which defines a behavior in terms of some other API rather than defining in terms of what it actually does, I would prefer two distinct modulus operators, and have the behavior clear in the name (e.g. modulusTruncate / modulusFloor).
shape as an operator, in addition to getter, is useful for chaining.
the sequence generation operator is useful.
- "range" is an odd name for this operator (even if other API's already made the sad faux pas), because ranges are an extent between low and high values, but this operator takes a range as input and generates a sequence (or ordered set) - it does not return a range.
- Omitting the explicit output shape will sadly resurrect the same floating-point issues we encountered last time we tried to add it, where dividing the limit by the delta could round one way or another around the edge cases.
This slice is inconsistent with the existing expression of starts and sizes. What happens if ends are less than starts? I can't say which form is truly better, as they're logically equivalent, but we should resolve them (possibly making the non-dynamic form take starts and ends, or making this new one takes starts and sizes). The same applies to this pad which doesn't match the existing pad operator. It would be good to consider convolution's expression ([beginningHeight, endingHeight, beginningWidth, endingWidth]), resample's expression (sizes or scales), pad's expression ([beginningPadding, endingPadding]), and slice, to be coherent where possible.

For those "dynamic" operators

Should these "dynamic" operators be prefixes or suffixes? One benefit to putting them as prefixes is that all the dynamic ones are near each other (if that's really a benefit), and a benefit of suffixes is that the operator starts with the more salient verb, and ordering them in a list like a table of contents puts related operators near each other alphabetically (reshape, reshapeDynamic, slice, sliceDynamic...). It's sad we can't just overload them rather than needing distinct names, but Javascript doesn't support that directly.

this "pad-to-chunks" algorithm is represented

It's worth considering how many of these shape computations actually need to be in the model itself (and whether model authors just did it there because it was possible), or whether its expressible as a graph input tensor attribute. Most commonly (as found in Stable Diffusion), the model just needs the inputs rounded up to some multiple. So a possibility is, kinda like how you're thinking of expressing a valid MLDynamicDimension range (min, max), you could also add a valid base multiple value which WebNN would massage accordingly. It's really overkill to compute these tiny shape computations on a powerful device, and you definitely don't want to compute them on a GPU only to read them back and stall. These shape computations really should be done on the CPU side somewhere in the pipeline, with either logic in WebNN or repeated logic in every backend (like the DML EP and CUDA EP heuristics which had some special code detecting the size of the tensor and which operator's the output was connected to, when deciding to prefer to keep the logic locally).

I did a dynamic shape analysis for the S3-DiT transformer model in Z-Image Turbo (details here).

how many of these shape computations actually need to be in the model itself

All shape computations in the model need to be in the model, take the "pad-to-chunks" as example. This is the only genuinely data-dependent pattern — it computes (32 - seq_len % 32) % 32, pads the tensor, creates a position mask via Range→ScatterND, then fills pad positions with a learned embedding. It's irreducible model logic, not shape routing.

For the other patterns, these are shape routing — but the target shapes depend on runtime input dimensions (batch size, sequence length, image height×width, etc).

ORT already has an optimizer pass (reshape_fusion.cc) that collapses several reshape chains into the most compact static representation possible — a constant shape tensor using 0 ("copy dim from input") and -1 ("infer from total element count"). For example in a Qwen2 SLM model:

Before fusion (6 nodes):
  Shape(input_ids)         → [batch_size, sequence_length]
  Gather(shape, idx=1)     → sequence_length (scalar)
  Unsqueeze(scalar, 0)     → [sequence_length]
  Concat([-1], [seq_len])  → [-1, sequence_length]
  Reshape(position_ids, concat_output)

After fusion (constant shape tensor):
  Reshape(position_ids, [-1, 0])

The shape routing patterns in the Z-Image transformer model are more complex (e.g., Reshape(data, [H/2 × W/2, 64]) where H and W come from the image input), but these also reduce to -1 since there's only one non-constant dim: Reshape(data, [-1, 64]). Ideally we can rely on frameworks' optimizer passes to fuse them — except for cases that can't be expressed with a single -1 (only one is allowed per reshape).

So I'd propose WebNN's reshape() support 0 and -1 in its newShape parameter. CoreML and ONNX support both 0 and -1, while TFLite only supports -1, but I suppose we can solve it via a trivial build-time substitution (since it knows the input operand shape).

@fdwr

For those "dynamic" operators, the output operand sizes are unknown at the build-time and cannot be validated against limits. As the next step, we plan to explore the dispatch-time shape validation when all tensors' sizes are specified or can be inferred.

We've implemented a prototype of dispatch-time shape validation (Verify that the intermediate and output shapes generated throughout the entire graph by actual inputs are valid):

At dispatch-time, when all input tensor sizes are known we clone the graph's operands and substitute DynamicDimension values with concrete uint32 values from the actual input tensors.
We perform a forward pass over the graph's operators (in topological order) that combines two parts (Shape inference and shape folding are integrated in one topological traversal):
- Shape Inference — For regular operators (conv2d, matmul, binary/unary/reduce ops, transpose, etc.), we compute the concrete output shape from the known input shapes using the same Validate*AndInferOutput functions used at build time, but with concrete uint32 values instead of DynamicDimension.
- Shape (constant) Folding — For dynamic operators whose output shape depends on a runtime tensor value (e.g., dynamicReshape's newShape operand), we trace backward through the producing subgraph (typically Shape → Gather → Concat patterns) to compute the concrete shape values.

dispatch(inputs, outputs):
  1. Build `dim_name` to `concrete_value` map from input tensors
  2. Clone ops, substitute `DynamicDimension` to `uint32`
  3. Forward pass over all ops (write back output shapes into cloned operands):
       Regular ops: `Validate*AndInferOutput` use concrete inputs
       Dynamic* ops: `ShapeFoldingInterpreter.Evaluate(shape_operand)` → folded values
                                compute output use folded values
  4. Validate calculated graph output shapes against user provided output shapes

The prototype is working well with end-to-end samples like “On-device OCR” and “Z-Image-Turbo” and we collect the performance data for validate time:

Model	Op num (w/o constant)	Input Sizes	Inference (ms)	Validate (ms)	Ratio
ch_PP-OCRv5_det (OCR)	664 (364)	[1, 3, 960, 960]	30.15	0.83	2.75%
ch_PP-OCRv5_rec (OCR)	746 (404)	[1, 3, 48, 192]	4.96	0.78	15.73%
transformer (Z-Image Turbo)	3080 (1769)	[1, 16, 1, 64, 64], [1, 113, 2560]	9846.596	22.772	0.23%
vae_decoder (Z-Image Turbo)	501 (376)	[1, 16, 1, 64, 64]	1482.583	1.846	0.12%

We can observe that validation time depends on only the graph structure (specifically the number of ops and the complexity of shape constant folding). The time complexity approaches O(N) where N = number of ops. Each op is visited once during the forward pass. Shape folding adds at most O(M) recursive evaluations where M = size of shape subgraphs (typically small constant-depth trees).

const knownShapeGraph = await potentiallyDynamicShapeGraph.computeShapes(inputs, outputs);

@fdwr previously suggested a computeShapes intermediate step between build and dispatch. Our current prototype embeds shape validation directly within dispatch — the inferred output shapes are used internally for validation but are not exposed to the caller.
We've encountered a real case where exposing them would be valuable: in the PP-OCRv5 rec model, the output contains dynamic dimensions that are not present in any input. When user calls session.run(feeds) (ORT Web WebNN EP), it cannot determine the output tensor sizes upfront. Also, when EP partitions a model into subgraphs, a subgraph's outputs may similarly contain dynamic dimensions not present in that subgraph's inputs (derived through internal operations). These creates a challenge for EP implementation. We'd like to discuss whether this should be part of the spec.

@Honry

@Honry

So I'd propose WebNN's reshape() support 0 and -1 in its newShape parameter. CoreML and ONNX support both 0 and -1, while TFLite only supports -1, but I suppose we can solve it via a trivial build-time substitution (since it knows the input operand shape).

Can we compose the equivalent of 0 and -1 with the newly proposed shape and dynamicReshape (or reshapeDynamic)?

For 0 (copy from input): shape(input) → gather(dim_index) → use in newShape tensor for dynamicReshape

For -1 (infer): shape(input) → reduceProd → div(known_dims_product) → insert into newShape tensor → dynamicReshape

@miaobin

@miaobin

Shape (constant) Folding — For dynamic operators whose output shape depends on a runtime tensor value (e.g., dynamicReshape's newShape operand), we trace backward through the producing subgraph (typically Shape → Gather → Concat patterns) to compute the concrete shape values.

Question: should the shape folding only allow shape() ops and constants as leaves in the producing subgraph? If graph inputs are also allowed as leaves, the tensor might reside on GPU, requiring a readback to CPU for shape inference — which could introduce a GPU→CPU sync and negate the benefit of async dispatch.

@huningxin

Can we compose the equivalent of 0 and -1 with the newly proposed shape and dynamicReshape (or reshapeDynamic)?

For 0 (copy from input): shape(input) → gather(dim_index) → use in newShape tensor for dynamicReshape

For -1 (infer): shape(input) → reduceProd → div(known_dims_product) → insert into newShape tensor → dynamicReshape

@huningxin, yes, that's possible, but with two drawbacks:

Decomposition overhead. Frameworks like ORT already fuse some patterns likeShape→Gather→Concat→Reshape into a single Reshape with a constant [0, -1, ..., ...] target shape. The WebNN EP must decompose that fused constant back into a shape-computation subgraph.
Opaque operand values. When newShape is an MLOperand, its values are not visible at graph-build time. How would the implementation resolve 0 or -1 at execution time is a challenge.

Question: should the shape folding only allow shape() ops and constants as leaves in the producing subgraph?

Yes, you are right and in our current POC, shape folding only succeeds when the producing subgraph's leaves are constants or shape() ops (which read dimension data, not tensor data). If a leaf is a graph input tensor, the prediction will return null and folding fails.

If graph inputs are also allowed as leaves, the tensor might reside on GPU, requiring a readback to CPU for shape inference.

In the models we've tested (Z-Image-Turbo, PP-OCRv5), new_shape operands are always derived from constants and shape ops (shape of already inferred tensors), so this works. However, if a model uses a graph input tensor directly as a shape operand, our current POC cannot resolve it without a readback.

We have drafted an explainer #945 to summarize the discussion on the current issue. And welcome further discussion and feedback on the explainer.

RESOLUTION: The group will merge the Dynamic Shape Explainer PR after addressing final review comments and proceeds to implementation experience and spec PR phase next. (issue #883)

Navigation Menu

Repository navigation

Support flexible input sizes #883

Testing:

Image classification MobileNet in webnn-samples

Text embedding all-MiniLM-L6-v2 model (based on @tarekziade 's webnn-graph)

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support flexible input sizes #883

Description

Activity

fdwr commented on Aug 28, 2025

huningxin commented on Aug 29, 2025

fdwr commented on Sep 4, 2025

mtavenrath commented on Oct 9, 2025

anssiko commented on Nov 27, 2025

ibelem commented on Dec 22, 2025

Testing:

tarekziade commented on Jan 9, 2026

huningxin commented on Jan 11, 2026

tarekziade commented on Jan 12, 2026

huningxin commented on Jan 29, 2026

Image classification MobileNet in webnn-samples

Text embedding all-MiniLM-L6-v2 model (based on @tarekziade 's webnn-graph)

huningxin commented on Feb 11, 2026

anssiko commented on Feb 12, 2026

5 remaining items

huningxin commented on Apr 23, 2026

fdwr commented on May 7, 2026

Honry commented on May 19, 2026

miaobin commented on May 19, 2026

huningxin commented on May 19, 2026

huningxin commented on May 19, 2026

Honry commented on May 20, 2026

miaobin commented on May 20, 2026

miaobin commented on Aug 7, 2026

anssiko commented on Sep 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions