rk3588设备上部署fsmn-vad并使用NPU加速。

[baichuan1997]

Notice: In order to resolve issues more efficiently, please raise issue following the template.
（注意：为了更加高效率解决您遇到的问题，请按照模板提问，补充细节）

❓ Questions and Help

Before asking:

1. search the issues.
2. search the docs.

What is your question?

如何在rk3588设备上部署fsmn-vad并使用NPU加速？
模型转换时候报错。
ValueError: The input shape [1, 'feats_length', 400] of 'speech' is not support!
E load_onnx: Please set the 'inputs' / 'input_size_list' parameters of 'rknn.load_onnx', or set the 'dyanmic_input' parameter of 'rknn.config' to fix the input shape!

Code

import cv2
import numpy as np

from rknn.api import RKNN
import os

if name == 'main':

platform = 'rk3588'
Width = 640
Height = 640
#Height = 640
# Model from https://github.com/airockchip/rknn_model_zoo
MODEL_PATH = './onnx_models/vad.onnx'
exp = MODEL_PATH.split('/')[-1].split('.')[0]
NEED_BUILD_MODEL = True
# NEED_BUILD_MODEL = False
# im_file = './dog_bike_car_640x640.jpg'

# Create RKNN object
rknn = RKNN()

OUT_DIR = "rknn_models"
RKNN_MODEL_PATH = './{}/{}_{}.rknn'.format(
    OUT_DIR, exp+'-'+str(Width)+'-'+str(Height), platform)
if NEED_BUILD_MODEL:
    # DATASET = '/home/share/datasets/imgs/subset/wg_subset_20.txt',
    #
    # DATASET = 'dataset.txt'
    rknn.config(mean_values=[[0, 0, 0]], std_values=[
                [255, 255, 255]], target_platform=platform, quantized_algorithm="normal", quantized_method="channel", optimization_level=1,)


    # Load model
    print('--> Loading model')
    # input_size_list = [(1, 1, 1, 3200, 1)]
    ret = rknn.load_onnx(MODEL_PATH)
    if ret != 0:
        print('load model failed!')
        exit(ret)
    print('done')

    # Build model
    print('--> Building model')
    #ret = rknn.build(do_quantization=False, rknn_batch_size=4)
    ret = rknn.build(do_quantization=False)
    if ret != 0:
        print('build model failed.')
        exit(ret)
    print('done')

    # Export rknn model
    if not os.path.exists(OUT_DIR):
        os.mkdir(OUT_DIR)
    print('--> Export RKNN model: {}'.format(RKNN_MODEL_PATH))
    ret = rknn.export_rknn(RKNN_MODEL_PATH, simplify=True)
    if ret != 0:
        print('Export rknn model failed.')
        exit(ret)
    print('done')
else:
    ret = rknn.load_rknn(RKNN_MODEL_PATH)

rknn.release()

What have you tried?

What's your environment?

• OS (e.g., Linux):
• FunASR Version (e.g., 1.0.0):
• ModelScope Version (e.g., 1.11.0):
• PyTorch Version (e.g., 2.0.0):
• How you installed funasr (pip, source):
• Python version:
• GPU (e.g., V100M32)
• CUDA/cuDNN version (e.g., cuda11.7):
• Docker version (e.g., funasr-runtime-sdk-cpu-0.4.1)
• Any other relevant information:

[DakeQQ]

欢迎参考此仓库。该仓库提供了一个FSMN_VAD端到端的ONNX实现，包括 STFT 预处理和判断后处理，简单地输出帧状态为“Speaking”或“Silence”。此FSMN_VAD_ONNX在 RK3588 NPU 上应能很好且方便地运行。