Android app - Error - Attempted to resize a static tensor to a new shape at dimension 0

[adonnini]

My Android application fails with Attempted to resize a static tensor to a new shape at dimension 0 error. Please find the full logcat below.

The shape of input datasets in my model is not static. Specifically, the number of steps in any one sequence varies.

here is the code I use to define the input dataset for the model in the Android application:

			float[] flat = flatten(tmpData);
			final long[]  shapeArrDataPytorchFlattened = new long[]{tmpData.length, 4, 1};
			arrDataPytorch = Tensor.fromBlob(flat, shapeArrDataPytorchFlattened);

where 4 is the number of features and tmpData.length is the size of the input dataset (with n rows and 4 columns)

here is the code I use to run inference:

			try {
				Log.i(TAG, " - neuralNetworkloadAndRunPytorch - Abut to run inference --- ");
				outputTensor = mModule.forward(from(arrDataPytorch)).toTensor();
			} catch (Exception e) {
				Log.i(TAG, " - neuralNetworkloadAndRunPytorch - Inference FAILED --- ");
				throw new RuntimeException(e);
			}

when I run inference on my model processed with torchscript and processed using pytorch mobile. I produce the input dataset as follows:

		        final long[] shapeArrDataPytorchFlattened = new long[]{1, flat.length};   //USED FOR PYTORCH MOBILE
			arrDataPytorch = Tensor.fromBlob(flat, shapeArrDataPytorchFlattened);

and run inference as follows:

				mModule = LiteModuleLoader.load(moduleFileAbsoluteFilePath);
				outputTensor = mModule.forward(IValue.from(arrDataPytorch)).toTensor();

This works producing reasonable results.

I would appreciate any thoughts as to what is causing the problem, and how I might go about fixing it.

Thanks

LOGCAT

12-05 16:48:49.983: I/NeuralNetworkService(16887):  - NeuralNetworkServiceRunnable - neuralNetworkInputPreparationRunning - 1 - 0
12-05 16:48:49.983: I/NeuralNetworkService(16887):  - NeuralNetworkServiceRunnable - neuralNetworkLoadAndRunRunning - 0 - 0
12-05 16:48:49.983: I/NeuralNetworkService(16887):  - NeuralNetworkServiceRunnable - About to run neuralNetworkloadAndRun --- 
12-05 16:48:49.983: I/NeuralNetworkService(16887):  - neuralNetworkloadAndRunPytorch - Running - 
12-05 16:48:49.983: I/NeuralNetworkService(16887):  - neuralNetworkloadAndRunPytorch - locationInformationDir - /data/user/0/com.android.contextq/files/locationInformation/
12-05 16:48:49.983: I/NeuralNetworkService(16887):  - neuralNetworkloadAndRunPytorch - savedNetworkArchiveLength - 120669888
12-05 16:48:49.983: I/NeuralNetworkService(16887):  - neuralNetworkloadAndRunPytorch - Abut to load module --- 
12-05 16:48:50.067: I/ETLOG(16887): Model file /data/user/0/com.android.contextq/files/locationInformation/tfmodel_exnnpack.pte is loaded.
12-05 16:48:50.067: I/ETLOG(16887): Setting up planned buffer 0, size 23366800.
12-05 16:48:50.077: W/libc(16887): Access denied finding property "ro.hardware.chipname"
12-05 16:48:50.078: W/adbd(13666): timeout expired while flushing socket, closing
12-05 16:48:50.080: D/XNNPACK(16887): allocated 6144 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.080: D/XNNPACK(16887): created workspace of size 774176
12-05 16:48:50.081: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.085: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.088: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.092: D/XNNPACK(16887): created workspace of size 387104
12-05 16:48:50.092: D/XNNPACK(16887): reusing tensor id #4 memory for tensor id #3 Node #2 Softmax
12-05 16:48:50.092: D/XNNPACK(16887): created workspace of size 42368
12-05 16:48:50.092: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.096: I/XNNPACK(16887): fuse Clamp Node #1 into upstream Node #0
12-05 16:48:50.097: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.113: D/XNNPACK(16887): allocated 4196352 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.127: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.130: D/XNNPACK(16887): created workspace of size 387104
12-05 16:48:50.130: D/XNNPACK(16887): reusing tensor id #4 memory for tensor id #3 Node #2 Softmax
12-05 16:48:50.130: D/XNNPACK(16887): created workspace of size 42368
12-05 16:48:50.130: I/XNNPACK(16887): fuse Clamp Node #1 into upstream Node #0
12-05 16:48:50.132: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.146: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.150: D/XNNPACK(16887): created workspace of size 387104
12-05 16:48:50.150: D/XNNPACK(16887): reusing tensor id #4 memory for tensor id #3 Node #2 Softmax
12-05 16:48:50.150: D/XNNPACK(16887): created workspace of size 42368
12-05 16:48:50.150: I/XNNPACK(16887): fuse Clamp Node #1 into upstream Node #0
12-05 16:48:50.152: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.166: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.170: D/XNNPACK(16887): created workspace of size 387104
12-05 16:48:50.170: D/XNNPACK(16887): reusing tensor id #4 memory for tensor id #3 Node #2 Softmax
12-05 16:48:50.170: D/XNNPACK(16887): created workspace of size 42368
12-05 16:48:50.170: I/XNNPACK(16887): fuse Clamp Node #1 into upstream Node #0
12-05 16:48:50.172: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.186: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.190: D/XNNPACK(16887): created workspace of size 387104
12-05 16:48:50.190: D/XNNPACK(16887): reusing tensor id #4 memory for tensor id #3 Node #2 Softmax
12-05 16:48:50.190: D/XNNPACK(16887): created workspace of size 42368
12-05 16:48:50.190: I/XNNPACK(16887): fuse Clamp Node #1 into upstream Node #0
12-05 16:48:50.192: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.206: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.209: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.213: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.217: D/XNNPACK(16887): allocated 8192 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.217: D/XNNPACK(16887): created workspace of size 1327136
12-05 16:48:50.217: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.221: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.224: D/XNNPACK(16887): reusing tensor id #8 memory for tensor id #5 Node #2 Softmax
12-05 16:48:50.224: D/XNNPACK(16887): created workspace of size 42368
12-05 16:48:50.225: D/XNNPACK(16887): created workspace of size 663584
12-05 16:48:50.225: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.229: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.232: I/XNNPACK(16887): fuse Clamp Node #2 into upstream Node #1
12-05 16:48:50.234: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.249: D/XNNPACK(16887): allocated 4196352 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.263: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.269: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.273: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.276: D/XNNPACK(16887): created workspace of size 387104
12-05 16:48:50.277: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.281: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.284: I/XNNPACK(16887): fuse Clamp Node #1 into upstream Node #0
12-05 16:48:50.286: D/StNfcHal(979): (#0C838) Rx 60 07 01 e2 
12-05 16:48:50.286: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.301: D/XNNPACK(16887): allocated 4196352 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.315: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.319: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.322: D/XNNPACK(16887): created workspace of size 663584
12-05 16:48:50.323: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.327: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.331: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.334: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.338: D/XNNPACK(16887): created workspace of size 387104
12-05 16:48:50.338: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.342: I/XNNPACK(16887): fuse Clamp Node #1 into upstream Node #0
12-05 16:48:50.344: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.357: D/XNNPACK(16887): created workspace of size 663584
12-05 16:48:50.358: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.361: D/XNNPACK(16887): created workspace of size 387104
12-05 16:48:50.362: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.365: I/XNNPACK(16887): fuse Clamp Node #1 into upstream Node #0
12-05 16:48:50.367: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.381: D/XNNPACK(16887): created workspace of size 663584
12-05 16:48:50.381: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.385: D/XNNPACK(16887): created workspace of size 387104
12-05 16:48:50.385: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.389: I/XNNPACK(16887): fuse Clamp Node #1 into upstream Node #0
12-05 16:48:50.390: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.404: D/XNNPACK(16887): created workspace of size 663584
12-05 16:48:50.405: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.408: D/XNNPACK(16887): created workspace of size 387104
12-05 16:48:50.409: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.412: I/XNNPACK(16887): fuse Clamp Node #1 into upstream Node #0
12-05 16:48:50.414: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.427: D/XNNPACK(16887): created workspace of size 663584
12-05 16:48:50.428: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.431: D/XNNPACK(16887): created workspace of size 387104
12-05 16:48:50.432: D/XNNPACK(16887): allocated 1050624 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.435: I/XNNPACK(16887): fuse Clamp Node #1 into upstream Node #0
12-05 16:48:50.437: D/XNNPACK(16887): allocated 4202496 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.450: D/XNNPACK(16887): allocated 16416 bytes for packed weights in Fully Connected (NC, F32) operator
12-05 16:48:50.467: I/NeuralNetworkService(16887):  - neuralNetworkloadAndRunPytorch - Abut to run inference --- 
12-05 16:48:50.467: I/ETLOG(16887): Attempted to resize a static tensor to a new shape at dimension 0 old_size: 27 new_size: 12716
12-05 16:48:50.467: I/ETLOG(16887): Error setting input 0: 0x10
12-05 16:48:50.467: I/ETLOG(16887): In function forward(), assert failed: set_input_status == Error::Ok
12-05 16:48:50.467: A/libc(16887): Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 16905 (Thread-2), pid 16887 (lNetworkService)
12-05 16:48:50.635: I/crash_dump64(17226): obtaining output fd from tombstoned, type: kDebuggerdTombstoneProto

[cccclai]

The difference between Lite Interpreter (Pytorch Mobile) and ExecuTorch is that, in Executorch, we plan memory ahead of time, which can help us to re-use/reduce memory usage in runtime. What is the dynamic part in the origin pytorch model? Can the dynamic part be upper bound?

Reference doc: https://pytorch.org/executorch/stable/compiler-memory-planning.html

[adonnini]

@cccclai Thanks for the response. Please bear with me as I am a beginner with Pytorch (my background is java and Android application development).

I did not create the model I am using (https://github.com/sharonrichushaji/trajectory-prediction-transformers/tree/master). I modified slightly to accommodate my datasets (produced by my Android application).

When you ask about the dynamic part of the model, could you please clarify? Dynamic with respect to which variables/parts of the model?

The model is an attention-based Transformer Network. Depending on what you mean by dynamic, being an encoder/decoder model the input from one part of the model to another is dynamic.

I read the document you referenced in your message. Would it make sense for me to use the
alloc_graph_input=False option? If I did, it's not clear what I should do in my application to do this:
"If the IO is not planned then users will be expected to provide data buffers to back these values at runtime"
Did I misunderstand the documentation?

Thanks

[kimishpatel]

If the IO is not planned then users will be expected to provide data buffers to back these values at runtime

What this means is that the inputs you provide will be a) copied to the memory planned for it during memory planning pass if IO was part of memory planning OR b) will not be copied since memory planning did not plan for this.

By default IO is planned and hence if follow this, https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/jni/jni_layer.cpp#L345, you will see that the output returned from the executor is references directly and there is a comment on the lifetime of the pointer referenced by the output tensor.

Now with respect to dynamic size. THere are a couple of things.

1. Export + executorch support upper bound dynamic size. That is you can tag the input tensors to have bounded dynamic size. E.g. On each dimension of the input what is the max size that dim can take. This helps export understand what is the maximum size expected on input, output and intermediate tensors. You can read more here https://pytorch.org/docs/stable/export.html#expressing-dynamism and @JacobSzwejbka is an expert on this who can give more input.
2. Do you know what part of your model is dynamic? You may not know this and thats fine. In that case I presume, you just expect that "I should be able to supply input of varying size", right?
3. Shape dynamism gets bit more complicated if the model you are interested in is also lowered to some delegate like say XNNPACK. In that case the delegate also need to support dynamic shape. For XNNPACK for example, this is not the case yet, so wont quite be able to run model via XNNPACK that can take inputs of different shapes.

[adonnini]

@kimishpatel Thanks!

a) It looks like I may have a problem as XNNPACK does not support shape dynamism as input sequences in the model I use are of varying length.

b) Your assumption is correct. I expected to be able to provide input of varying size

c) Given what @cccclai said in his comment regarding the difference between pytorch mobile and executorch, and also based on what you said in your point 1. above, I will try the torch.export.dynamic_dim() API.

d) Why is delegation (e.g. via XNNPACK) necessary in order to lower a model onto an edge device using executorch? Sorry for bringing it up again. With pytorch mobile it was not (unless I am mistaken). Is there an alternative to using XNNPACK?

Thanks

[cccclai]

Why is delegation (e.g. via XNNPACK) necessary in order to lower a model onto an edge device using executorch?

Delegation is for delegate part of or the whole model to some powerful backends on device. Different edge devices may have different backends. XNNPACK(https://github.com/google/XNNPACK) is the one of the most powerful backends on CPU. For example, on iOS there will be some other powerful backends (https://github.com/pytorch/executorch/tree/main/backends/apple) like coreml and mps. Qualcomm chipset might too.

In PyTorch Mobile, XNNPACK is pretty much like a default backend and it runs after we call optimized_for_mobile. In PyTorch Mobile, we also have limited backends available on PyTorch Mobile like coreml (https://pytorch.org/tutorials/prototype/ios_coreml_workflow.html)

[adonnini]

@cccclai Thanks.
When I used pytorch mobile I chose to not optimize before lowering the model to the edge device. Pytorch mobile gave me that option, if I am not mistaken.
Executorch does not give you the option to not optimize, right?
By the way, I chose to not use optimization in order to first get the basic mechanism working (i.e. perform inference on the edge device successfully).