QNN OP profiling

[MrJungle1]

🚀 The feature, motivation and pitch

Is there any way to sample and analyze the time consumption of each op in the qnn backend?

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @Gasoonjia @cccclai @cbilgin

[guangy10]

cc: @cccclai @Gasoonjia

[haowhsu-quic]

Hi @MrJungle1, please check following minimal example to gather profiling result. We'll update the document as well.

import copy, os, torch
 
from torchao.quantization.pt2e.quantize_pt2e import (
    convert_pt2e,
    prepare_pt2e,
)
from executorch.backends.qualcomm.utils.utils import (
    dump_context_from_pte,
    generate_htp_compiler_spec,
    generate_qnn_executorch_compiler_spec,
    QcomChipset,
    to_edge_transform_and_lower_to_qnn,
)
from executorch.backends.qualcomm.tests.models import SimpleModel
from executorch.examples.qualcomm.utils import make_quantizer, SimpleADB
from executorch.devtools import generate_etrecord, Inspector
from executorch.devtools.inspector._inspector_utils import TimeScale
 
if __name__ == '__main__':
    # capture nn.Module into ExportedProgram
    sample_input = (torch.randn(1, 32, 28, 28), torch.randn(1, 32, 28, 28))
    model = torch.export.export(SimpleModel(), sample_input).module()
     
    # generate QnnQuantizer
    quantizer = make_quantizer()
    prepared = prepare_pt2e(model, quantizer)
    # perform calibration
    prepared(*sample_input)
    converted = convert_pt2e(prepared)
     
    # setup compile spec for HTP backend
    backend_options = generate_htp_compiler_spec(use_fp16=False)
    compiler_specs = generate_qnn_executorch_compiler_spec(
        soc_model=QcomChipset.SM8750,
        backend_options=backend_options,
        profile=True,
    )
    # lower to QNN ExecuTorch Backend
    edge_prog_mgr = to_edge_transform_and_lower_to_qnn(
        module=converted,
        inputs=sample_input,
        compiler_specs=compiler_specs,
    )
    # for inspector API
    edge_copy = copy.deepcopy(edge_prog_mgr)
    # store pte file
    exec_prog = edge_prog_mgr.to_executorch()
    pte_name = "simple_example.pte"
    with open(pte_name, "wb") as f:
        exec_prog.write_to_file(f)
 
    # setup ADB for on-device execution
    adb = SimpleADB(
        qnn_sdk=os.getenv("QNN_SDK_ROOT"),
        build_path="build-android",
        pte_path=pte_name,
        workspace="/data/local/tmp/simple_example",
        device_id="5f396958",
        soc_model="SM8750",
    )
    input_list = "input_0_0.raw input_0_1.raw\n"
    adb.push(inputs=[sample_input], input_list=input_list)
    adb.execute()
 
    # pull etdump back and display the statistics
    adb.pull_etdump(".")
    generate_etrecord("etrecord.bin", edge_copy, exec_prog)
    inspector = Inspector(
        etdump_path="etdump.etdp",
        etrecord="etrecord.bin",
        source_time_scale=TimeScale.CYCLES,
        target_time_scale=TimeScale.CYCLES,
    )
    df = inspector.to_dataframe()
    # here we only dump the first 15 rows
    print(df.head(15))

You should see profiling results like following:

[MrJungle1]

Thank you very much for your reply. I have now generated similar results to yours, but I want to know what these mean. I printed all the lines here, why are the convolutions all 0.0? Isn't this the most time-consuming?

[haowhsu-quic]

Hi @MrJungle1, QNN HTP will perform some pattern matching and fuse ops into more performant one.
In this case, the convolution and relu will be fused together. You will see the cycles appear in relu only.

[MrJungle1]

@haowhsu-quic Thank you for your patient reply!!! I would also like to ask if dynamic shapes are not currently supported. I tried to add dynamic_shapes when exporting SimpleModel. The input used in actual machine reasoning was 32, 64, 64, and it was directly Aborted. Below is my code.
sample_input = (torch.randn(1, 32, 28, 28), torch.randn(1, 32, 28, 28))
dynamic_shapes = {
"x": {
2: torch.export.Dim("h", min=16, max=1024),
3: torch.export.Dim("w", min=16, max=1024),
},
"y": {
2: torch.export.Dim("h", min=16, max=1024),
3: torch.export.Dim("w", min=16, max=1024),
}
}
model = torch.export.export(SimpleModel(), sample_input, dynamic_shapes=dynamic_shapes).module()

Actual input：
sample_input = (torch.randn(1, 32, 64, 64), torch.randn(1, 32, 64, 64))
input_list = "input_0_0.raw input_0_1.raw\n"
adb.push(inputs=[sample_input], input_list=input_list)

8gen3 result

[MrJungle1]

To add, I have passed expected_input_shape and expected_output_shape in SimpleADB,
for example, expected_input_shape=(torch.Size([1, 32, 64, 64]),torch.Size([1, 32, 64, 64]))
expected_input_shape=(torch.Size([8,10]),)

[MrJungle1]

Sorry! I made a mistake just now. Now it works, but when I try to use the model in the tutorial, to_edge_transform_and_lower_to_qnn reports an error when exporting. Are there still some ops that do not support dynamic shapes?

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.seq = torch.nn.Sequential(
            torch.nn.Conv2d(1, 8, 3),
            torch.nn.ReLU(),
            torch.nn.Conv2d(8, 16, 3),
            torch.nn.ReLU(),
            torch.nn.AdaptiveAvgPool2d((1,1))
        )
        self.linear = torch.nn.Linear(16, 10)
    
    def forward(self, x):
        y = self.seq(x)
        y = torch.flatten(y, 1)
        y = self.linear(y)
        return y
    model = Model()
    inputs = (torch.randn(1,1,16,16),)
    dynamic_shapes = {
        "x": {
            2: torch.export.Dim("h", min=16, max=1024),
            3: torch.export.Dim("w", min=16, max=1024),
        }
    } 
model = torch.export.export(Model(), inputs, dynamic_shapes=dynamic_shapes).module()
    backend_options = generate_htp_compiler_spec(use_fp16=True)
    compiler_specs = generate_qnn_executorch_compiler_spec(
        soc_model=QcomChipset.SM8650,
        backend_options=backend_options,
        profile=True,
    )
    # lower to QNN ExecuTorch Backend
    edge_prog_mgr = to_edge_transform_and_lower_to_qnn(
        module=model,
        inputs=inputs,
        compiler_specs=compiler_specs,
        dynamic_shapes=dynamic_shapes,
    )

Execution to to_edge_transform_and_lower_to_qnn reported an error,

WARNING:root:Op aten.unbind.int was requested for preservation by partitioner. This request is ignored because it is in a blocklist.
WARNING:root:Op aten.unbind.int was requested for preservation by partitioner. This request is ignored because it is in a blocklist.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[WARNING] [Qnn ExecuTorch]: Initializing HtpProvider

[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[WARNING] [Qnn ExecuTorch]: Qnn API version 2.27.0 is used. The version is tested against 2.23.0.
[INFO] [Qnn ExecuTorch]: Caching: Caching is in SAVE MODE.
[WARNING] [Qnn ExecuTorch]: Performance Estimates unsupported

[WARNING] [Qnn ExecuTorch]: Arch 68 set by custom config is different from arch associated with SoC 57, will overwrite it to 75

[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
[WARNING] [Qnn ExecuTorch]: Sanitizing the value for hvx_threads and setting to default

[QNN Partitioner Op Support]: aten.linear.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: Tensor incorrectly has dynamic dimensions.

[ERROR] [Qnn ExecuTorch]: Op config was not well formed.

[ERROR] [Qnn ExecuTorch]: validateNativeOps master op validator aten_adaptive_avg_pool2d_default:qti.aisw:PoolAvg2d failed 3110

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_adaptive_avg_pool2d_default with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.adaptive_avg_pool2d.default | False
[QNN Partitioner Op Support]: aten.relu.default | True
[QNN Partitioner Op Support]: aten.convolution.default | True
[QNN Partitioner Op Support]: aten.relu.default | True
[QNN Partitioner Op Support]: aten.convolution.default | True
INFO:executorch.backends.qualcomm.partition.qnn_partitioner:Qnn partitioner will delegate torch mutable buffer with the same I/O address during the runtime, so if your model contains mutable buffer, then you can get the better performance with skip_mutable_buffer=False. If you encounter accuracy issue during the runtime, then please set skip_mutable_buffer=True and try again.
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[WARNING] [Qnn ExecuTorch]: Qnn API version 2.27.0 is used. The version is tested against 2.23.0.
[INFO] [Qnn ExecuTorch]: Caching: Caching is in SAVE MODE.
[WARNING] [Qnn ExecuTorch]: Performance Estimates unsupported

[WARNING] [Qnn ExecuTorch]: Arch 68 set by custom config is different from arch associated with SoC 57, will overwrite it to 75

[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
[WARNING] [Qnn ExecuTorch]: Sanitizing the value for hvx_threads and setting to default

INFO:executorch.backends.qualcomm.qnn_preprocess:Processing Method(0): (1/2)
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_permute_copy_default, aten.permute_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_convolution_default, aten.convolution.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_relu_default, aten.relu.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_convolution_default_1, aten.convolution.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_relu_default_1, aten.relu.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_permute_copy_default_1, aten.permute_copy.default
[WARNING] [Qnn ExecuTorch]: op_registry_prepare.cc:74:WARNING:Selecting disabled op for 0x10cb0000001a q::ConvLayer.opt.weights_to_vtcm_dyn

[ERROR] [Qnn ExecuTorch]: graph_prepare.cc:220::ERROR:could not create op: q::ConvLayer.opt.weights_to_vtcm_dyn

[ERROR] [Qnn ExecuTorch]: graph_prepare.cc:222::ERROR:Op creation failure, op id=0x10cb0000001a (q::ConvLayer.opt.weights_to_vtcm_dyn) total_inputs=1

[ERROR] [Qnn ExecuTorch]: graph_prepare.cc:208: Input 0: id=[0x10db0000001a] op=[Const] output0=[14ConcreteTensorIN5Tdefs12pkWeightsF16EE]

[ERROR] [Qnn ExecuTorch]: graph_prepare.cc:1574::ERROR:Op 0x10cb0000001a preparation failed with err:-1

[ERROR] [Qnn ExecuTorch]: "aten_relu_default" generated: could not create op

[ERROR] [Qnn ExecuTorch]: RouterX86 graph prepare failed 12

[ERROR] [Qnn ExecuTorch]: Failed to finalize graph (id: 1) with err 1002

[ERROR] [Qnn ExecuTorch]: Failed to finalize Qnn Graph with error: 1002
[ERROR] [Qnn ExecuTorch]: Fail to compile QNN graph
Traceback (most recent call last):
File "/workspace/executorch_use/test_profling.py", line 86, in
edge_prog_mgr = to_edge_transform_and_lower_to_qnn(
File "/workspace/executorch/backends/qualcomm/utils/utils.py", line 438, in to_edge_transform_and_lower_to_qnn
return to_edge_transform_and_lower(
File "/workspace/executorch/exir/program/_program.py", line 114, in wrapper
return func(self, *args, **kwargs)
File "/workspace/executorch/exir/program/_program.py", line 1262, in to_edge_transform_and_lower
edge_manager = edge_manager.to_backend(method_to_partitioner)
File "/workspace/executorch/exir/program/_program.py", line 114, in wrapper
return func(self, *args, **kwargs)
File "/workspace/executorch/exir/program/_program.py", line 1533, in to_backend
new_edge_programs = to_backend(method_to_programs_and_partitioners)
File "/root/anaconda3/envs/executorch/lib/python3.10/functools.py", line 878, in wrapper
return dispatch(args[0].class)(*args, **kw)
File "/workspace/executorch/exir/backend/backend_api.py", line 761, in _
lower_all_submodules_to_backend(
File "/workspace/executorch/exir/backend/backend_api.py", line 595, in lower_all_submodules_to_backend
backend_name_to_subclass[backend_id].preprocess_multimethod(
File "/workspace/executorch/backends/qualcomm/qnn_preprocess.py", line 181, in preprocess_multimethod
assert (
AssertionError: Failed to generate Qnn context binary.
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend

[haowhsu-quic]

Hi @MrJungle1, currently there are limited ops can support dynamic shape. The PoolAvg2d is not supported yet, please visit Qualcomm AI Engine Direct doc to check the capability of each op.

[MrJungle1]

@haowhsu-quic Thank you very much for your patience in replying!!! It helped me a lot. Thank you again!!~~

[MrJungle1]

In addition, I found that some models require a lot of RAM + swap when compiled into pte. Sometimes my 16g RAM is completely killed. My model size is only this large. What is the situation? Why does it require such a large amount of RAM? Is there a way to reduce it?
-rw-r--r-- 1 root root 212M Jul 17 17:51 vocoder.pt2

[MrJungle1]

In addition, I found that some models require a lot of RAM + swap when compiled into pte. Sometimes my 16g RAM is completely killed. My model size is only this large. What is the situation? Why does it require such a large amount of RAM? Is there a way to reduce it? -rw-r--r-- 1 root root 212M Jul 17 17:51 vocoder.pt2

Also, When I finally generated the pte, I set 100 different inputs, but the phone restarted after running a few.

[MrJungle1]

This is my final output of generating pte