MPS and XPU support

[exdysa]

RE: #1011

It is a few simple lines that allow MPS and XPU to pass through gatekeeping, enabling GPU other than NVIDIA's to run.

Caches, device setting, and GPU assert statements have been changed, though possible I overlooked something.

Note: I am unsure of the lines in cosyvoice/cli/model.py and whether they need to be changed, this one because I am not familiar with tensorRT:
self.flow.decoder.estimator_engine = trt.Runtime(trt.Logger(trt.Logger.INFO)).deserialize_cuda_engine(f.read())

And this will assign nullcontext(), but I know no alternative:
67/336
self.llm_context = torch.cuda.stream(torch.cuda.Stream(self.device)) if torch.cuda.is_available() else nullcontext()

[e1732a364fed]

cosyvoice/cli/model.py 两个 mps 的地方少了引号

 elif self.device == "mps:

exdysa#1

[exdysa]

cosyvoice/cli/model.py 两个 mps 的地方少了引号

Thank you! Now type checked & linted. Needs some testing.

我没有安装这个，因为我不知道如何修复这两条线。如果你想和我一起测试，我可以进一步提供帮助。

self.flow.decoder.estimator_engine = trt.Runtime(trt.Logger(trt.Logger.INFO)).deserialize_cuda_engine(f.read())

self.llm_context = torch.cuda.stream(torch.cuda.Stream(self.device)) if torch.cuda.is_available() else nullcontext()

[jockiller]

非常感谢，可以在m4pro上正常运行，希望能维护下去

[exdysa]

在XPU和CUDA测试后，我希望它能够合并，因为这对所有团队来说都会减少麻烦......

Thank you for CosyVoice2. I have updated support for all cards and corrected seeding routines. Please test before merging.

[bli1348]

I have A770, I'd like to use your commits to enable xpu. but what packages do I need to install to work with your codes? Just torch xpu nightly? is there a requirements.txt ? or an install guide?

[exdysa]

I have A770, I'd like to use your commits to enable xpu.

Great! You should ensure you have this
https://www.intel.com/content/www/us/en/developer/articles/tool/pytorch-prerequisites-for-intel-gpu/2-7.html#driver-installation

and then simply :

pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/xpu

inside of your environment. Thats it!

If you have trouble make an issue request here and I'll do my best to patch it for you. Please do not make an issue for this in the FunAudioLLM repo. It's not their problem, they have yet to accept my help.

Unfortunately there aren't any formal instructions because there are not any formal instructions in the original CUDA installation. It is safe to presume the code authors consider it out of scope. There isn't a requirements.txt because setting up requirements for pip and GPUs currently does not work very well.

I'll work on including a setup.py file in this merge that will get the correct PyTorch for all cards.

[bli1348]

I don't see issue tab on your repo. I just finished install. try to run
python webui.py --port 50000 --model_dir pretrained_models/CosyVoice-300M
but got
(cosyvoice) PS D:\cosyvoice> python webui.py --port 50000 --model_dir pretrained_models/CosyVoice-300M
Traceback (most recent call last):
File "D:\cosyvoice\webui.py", line 25, in
from cosyvoice.cli.cosyvoice import CosyVoice, CosyVoice2
File "D:\cosyvoice\cosyvoice\cli\cosyvoice.py", line 21, in
from cosyvoice.cli.frontend import CosyVoiceFrontEnd
File "D:\cosyvoice\cosyvoice\cli\frontend.py", line 17, in
import onnxruntime
File "C:\Users\bb\miniconda3\envs\cosyvoice\lib\site-packages\onnxruntime_init_.py", line 61, in
raise import_capi_exception
File "C:\Users\bb\miniconda3\envs\cosyvoice\lib\site-packages\onnxruntime_init_.py", line 24, in
from onnxruntime.capi._pybind_state import (
File "C:\Users\bb\miniconda3\envs\cosyvoice\lib\site-packages\onnxruntime\capi_pybind_state.py", line 32, in
from .onnxruntime_pybind11_state import * # noqa
ImportError: DLL load failed while importing onnxruntime_pybind11_state: A dynamic link library (DLL) initialization routine failed.

[exdysa]

I don't see issue tab on your repo. I just finished install.

sorry about that, expect progress on this this to continue here : https://github.com/exdysa/CosyVoice/issues

try https://github.com/eighteen-k-gold-malow/CosyVoice-XPU/commits/main/ for the time being, i will look at working with them on this