-
Notifications
You must be signed in to change notification settings - Fork 1.2k
fix #2587: Resolve VAD multithreading issue #2613
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
# Conflicts: # runtime/onnxruntime/src/funasrruntime.cpp
# Conflicts: # runtime/onnxruntime/src/funasrruntime.cpp
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Summary of Changes
Hello @majic31, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request resolves a critical multithreading bug affecting the Voice Activity Detection (VAD) inference process. By correctly handling the cache
parameter to prevent shared mutable state across threads, it ensures that VAD can operate reliably in concurrent environments, significantly improving the stability and performance of the system when used in multithreaded applications.
Highlights
- Multithreading Bug Fix: Addresses a critical multithreading issue in the VAD (Voice Activity Detection) inference method. The problem stemmed from a shared mutable default
cache
dictionary, which led to conflicts when the method was called concurrently from multiple threads. - Parameter Default Change: The
cache
parameter in theinference
method offunasr/models/fsmn_vad_streaming/model.py
has been changed from having a default value of an empty dictionary ({}
) toNone
. - Independent Cache Initialization: New logic has been introduced within the
inference
method to initialize a fresh, independentcache
dictionary (cache = {}
) if thecache
parameter is not explicitly provided (i.e., it defaults toNone
). This ensures that each thread operates with its own distinct cache, resolving the concurrency issue.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command>
or @gemini-code-assist <command>
. Below is a summary of the supported commands.
Feature | Command | Description |
---|---|---|
Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/
folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request effectively resolves a critical multithreading issue in the VAD inference method caused by a mutable default argument. The fix, changing the default value of cache
to None
and initializing it within the function, is correct. I've added one suggestion to make the implementation more idiomatic and robust by using an explicit is None
check, which is a common best practice in Python.
ok Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
问题描述:
如 issue #2587 中提到,当在多线程环境中调用 VAD(语音活动检测)时,会导致错误。问题的根本原因是 VAD 推理方法中的 cache 参数默认使用了一个空字典 ({}),而字典是可变对象,在多线程环境中多个线程共享了同一个入参的cache,从而导致了缓存冲突的问题。
修复思路:
为了解决这个问题,cache 参数如果未传入,则默认值为 None。在方法内部,根据需要再创建一个新的空字典 ({}),这样每个线程都能使用独立的 dict,避免了线程间的冲突。
经过测试,修复后 VAD 多线程推理可以正常运行。
测试脚本:(修复前运行有异常,修复后可以正常运行)
`import traceback
from concurrent.futures import ThreadPoolExecutor, as_completed
from funasr import AutoModel
if name == 'main':
model = AutoModel(
model="iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch",
vad_model="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch",
punc_model="iic/punc_ct-transformer_zh-cn-common-vocab272727-pytorch",
# spk_model="iic/speech_campplus_sv_zh-cn_16k-common",
)
file_name_list = [
"https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav",
"https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav",
"https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav"
]
with ThreadPoolExecutor() as executor:
# futures = [executor.submit(modelscope_local.inference_vad_offline, file_name) for file_name in file_name_list]
futures = [executor.submit(model.generate, input=file_name) for file_name in file_name_list]
results = []
for future in as_completed(futures):
try:
results.append(future.result())
except Exception as e:
print(f"获取任务结果时出错: {traceback.format_exc()}")
print("\n===== 处理结果 =====")
for result in results:
print(f'result={result}')`