[IPEX][XPU][Windows 11] It takes forever to run the first pass

[whchan05]

Describe the issue

Title. Once the first pass is complete, subsequent passes can be run faster. Seem to only happen when the device is set to XPU.

XPU:

CPU:

Environment:

CPU: 5700X
GPU: A770LE
RAM: 96GB
OS: Win 11

oneAPI:

DPC++ Compiler: 2023.2.0
MKL: 2023.2.0
Torch: 2.0.0a0
IPEX: 2.0.110+gitba7f6c1

Driver:

31.0.101.4577 WHQL

Others:

Miniconda: 23.5.2
Python: 3.10.12
Stable Diffusion WebUI: https://github.com/jbaboval/stable-diffusion-webui Commit: 197dedd

[Mindset-Official]

takes about 5-10 minutes to start generating the first image in Sd.next for me in native windows. Also takes about 257.9s to startup. After it starts the first image I can start generating pretty quickly, when loading a new model it sometimes takes a few minutes but not as long as the first time and overtime it's actually fast like it should be. this may have something to do with lack of supported torchvision, but I'm not sure.

[jingxu10]

The wheel files were not compiled with AOT enabled, thus the first iteration will take longer time.
@min-jean-cho FYI.

[whchan05]

Just now I failed to compile from source again, would be great if prebuilt wheel files with AOT device specified is uploaded

Error Log

2 warnings generated. [1044/1049] Linking CXX static library csrc\gpu\oneDNN\src\dnnl.lib ignoring unknown argument: -fsycl ignoring unknown argument: -Wno-unknown-argument ignoring unknown argument: -Qoption,link,/machine:x64 [1048/1049] Linking CXX shared library csrc\gpu\intel-ext-pt-gpu.dll FAILED: csrc/gpu/intel-ext-pt-gpu.dll csrc/gpu/intel-ext-pt-gpu.lib cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_dll --intdir=csrc\gpu\CMakeFiles\intel-ext-pt-gpu.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100220~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100220~1.0\x64\mt.exe --manifests -- C:\oneAPI\compiler\2023.2.0\windows\bin\icx.exe /nologo @CMakeFiles\intel-ext-pt-gpu.rsp -LD /Qoption,link,/machine:x64 -rdynamic -Wl,-Bsymbolic-functions /INCREMENTAL:NO -fsycl /EHsc -fsycl-max-parallel-link-jobs=16 -fsycl-targets=spir64_gen,spir64 -fsycl-link-huge-device-code /Xs "-device ats-m150 -options ' -cl-intel-enable-auto-large-GRF-mode -cl-poison-unsupported-fp64-kernels'" -link /out:csrc\gpu\intel-ext-pt-gpu.dll /implib:csrc\gpu\intel-ext-pt-gpu.lib /pdb:csrc\gpu\intel-ext-pt-gpu.pdb /version:0.0 && cd ." LINK: command "C:\oneAPI\compiler\2023.2.0\windows\bin\icx.exe /nologo @CMakeFiles\intel-ext-pt-gpu.rsp -LD /Qoption,link,/machine:x64 -rdynamic -Wl,-Bsymbolic-functions /INCREMENTAL:NO -fsycl /EHsc -fsycl-max-parallel-link-jobs=16 -fsycl-targets=spir64_gen,spir64 -fsycl-link-huge-device-code /Xs -device ats-m150 -options ' -cl-intel-enable-auto-large-GRF-mode -cl-poison-unsupported-fp64-kernels' -link /out:csrc\gpu\intel-ext-pt-gpu.dll /implib:csrc\gpu\intel-ext-pt-gpu.lib /pdb:csrc\gpu\intel-ext-pt-gpu.pdb /version:0.0 /MANIFEST /MANIFESTFILE:csrc\gpu\intel-ext-pt-gpu.dll.manifest" failed (exit code 1120) with the following output: icx: warning: unknown argument ignored in clang-cl: '-rdynamic' [-Wunknown-argument] icx: warning: unknown argument ignored in clang-cl: '-fsycl-link-huge-device-code' [-Wunknown-argument] icx: warning: argument unused during compilation: '-EHsc' [-Wunused-command-line-argument] Creating library csrc\gpu\intel-ext-pt-gpu.lib and object csrc\gpu\intel-ext-pt-gpu.exp reducer-86aa6b.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) public: virtual void __cdecl c10d::Timer::record(enum c10d::Timer::Event)" (__imp_?record@Timer@c10d@@UEAAXW4Event@12@@Z) referenced in function "public: virtual void __cdecl c10d::`anonymous namespace'::XPUTimer::record(enum A0xBD354422::Timer::Event)" (?record@XPUTimer@?A0xBD354422@c10d@@UEAAXW4Event@Timer@2@@Z) Hint on symbols that are defined and could potentially match: "__declspec(dllimport) public: virtual void __cdecl c10::impl::DeviceGuardImplInterface::record(void * *,class c10::Stream const &,signed char,enum c10::EventFlag)const " (__imp_?record@DeviceGuardImplInterface@impl@c10@@UEBAXPEAPEAXAEBVStream@3@CW4EventFlag@3@@Z) reducer-86aa6b.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) class c10::Registry >,struct c10::Device> * __cdecl c10d::TimerRegistry(void)" (__imp_?TimerRegistry@c10d@@YAPEAV?$Registry@W4DeviceType@c10@@V?$unique_ptr@VTimer@c10d@@U?$default_delete@VTimer@c10d@@@std@@@std@@UDevice@2@@c10@@XZ) referenced in function _GLOBAL__sub_I_reducer_e9db2c.cpp reducer-86aa6b.obj : error LNK2001: unresolved external symbol "public: virtual void __cdecl c10d::Timer::record(enum c10d::Timer::Event)" (?record@Timer@c10d@@UEAAXW4Event@12@@Z) C:\Users\playe\AppData\Local\Temp\icx-71bffa\fusion_pass-b19d20.out : fatal error LNK1120: 3 unresolved externals icx: error: linker command failed with exit code 1120 (use -v to see invocation) ninja: build stopped: subcommand failed. Traceback (most recent call last): File "C:\Users\playe\bundle\intel-extension-for-pytorch\setup.py", line 1103, in setup( File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\__init__.py", line 107, in setup return distutils.core.setup(**attrs) File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\_distutils\core.py", line 185, in setup return run_commands(dist) File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands dist.run_commands() File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands self.run_command(cmd) File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\dist.py", line 1244, in run_command super().run_command(command) File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command cmd_obj.run() File "C:\Users\playe\miniconda3\lib\site-packages\wheel\bdist_wheel.py", line 325, in run self.run_command("build") File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command self.distribution.run_command(command) File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\dist.py", line 1244, in run_command super().run_command(command) File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command cmd_obj.run() File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\_distutils\command\build.py", line 131, in run self.run_command(cmd_name) File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command self.distribution.run_command(command) File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\dist.py", line 1244, in run_command super().run_command(command) File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command cmd_obj.run() File "C:\Users\playe\bundle\intel-extension-for-pytorch\setup.py", line 1072, in run self.run_command("build_clib") File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command self.distribution.run_command(command) File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\dist.py", line 1244, in run_command super().run_command(command) File "C:\Users\playe\miniconda3\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command cmd_obj.run() File "C:\Users\playe\bundle\intel-extension-for-pytorch\setup.py", line 783, in run _build_project(build_args, ipex_xpu_build_dir, my_env, use_ninja) File "C:\Users\playe\bundle\intel-extension-for-pytorch\setup.py", line 554, in _build_project check_call(["ninja"] + build_args, cwd=build_dir, env=build_env) File "C:\Users\playe\miniconda3\lib\subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['ninja', '-j', '16', 'install']' returned non-zero exit status 1.

[min-jean-cho]

Thanks @whchan05, @Mindset-Official. The first kernel run without AOT compilation is expected to take a little longer. AOT compilation will significantly reduce the first kernel run time. In the meantime, feel free to try building from source with AOT as described here.

[min-jean-cho]

Just now I failed to compile from source again, would be great if prebuilt wheel files with AOT device specified is uploaded

Thanks @whchan05, we will shortly investigate the issue.

[min-jean-cho]

Docker is not required for IPEX on native windows.

[Mindset-Official]

Thanks @whchan05, @Mindset-Official. The first kernel run without AOT compilation is expected to take a little longer. AOT compilation will significantly reduce the first kernel run time. In the meantime, feel free to try building from source with AOT as described here.

I have attempted to build from source with AOT but it takes a long time during the compile (about 6 hours) when reaching this point

[1046/1049] Linking CXX static library csrc\gpu\oneDNN\src\dnnl.lib
ignoring unknown argument: -fsycl
ignoring unknown argument: -Wno-unknown-argument
ignoring unknown argument: -Qoption,link,/machine:x64
[1047/1049] Linking CXX shared library csrc\gpu\intel-ext-pt-gpu.dll

I was able to build the wheel files for IPEX but then i run into this error

Python 3.10.12 | packaged by Anaconda, Inc. | (main, Jul 5 2023, 19:01:18) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import torch
import intel_extension_for_pytorch
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\mymin\miniconda3\lib\site-packages\intel_extension_for_pytorch_init_.py", line 100, in
from . import inductor
File "C:\Users\mymin\miniconda3\lib\site-packages\intel_extension_for_pytorch_inductor_init.py", line 1, in
from torch._inductor.codegen.common import register_backend_for_device
ImportError: cannot import name 'register_backend_for_device' from 'torch._inductor.codegen.common' (C:\Users\mymin\miniconda3\lib\site-packages\torch_inductor\codegen\common.py)

I am unsure if I am missing something or there is an issue with AOT in windows? thanks for the help.

my system is

win 11
amd 5600
arc a750
running on Sata ssd

[Vipitis]

consider compiling prebuilt wheels with AOT enabled, especially the windows native variant. It should be s much better experience, and you only need to support Arc dGPU. As PVC is not available as a workstation card and I suspect will not run on Windows (unless both of that will change in the near future).
calling first model inference takes slightly over 11 minutes on my old system.

[min-jean-cho]

I have attempted to build from source with AOT but it takes a long time during the compile (about 6 hours) when reaching this point

Thanks @Mindset-Official, we are currently aware of the long AOT build time. Even though AOT build is expected to take longer than non-AOT (i.e., JIT), this long build time (~6 hrs) on windows is unreasonable. This is under investigation.

[min-jean-cho]

Thanks @Vipitis for the feedback, we will take into account.

[Vipitis]

[...] building from source with AOT as described here.

found an error in the linked doc. first sentence claims the prebuilt wheels have AOT enabled or both GPU devices. Could you confirm what the wheels hosted on https://developer.intel.com/ipex-whl-stable-xpu are built with?

[jingxu10]

Hi, that sentence applies to wheel files for Linux. Wheel files for Windows don't have AOT enabled yet.
Thanks for pointing this out. We will amend the doc.