CUDA_PATH is set but CUDA wasnt able to be loaded

[Noor-Nizar]

Describe the issue

Below is an error I'm getting trying to use the exported model in onnx format,

1 - Training and infering on GPU using pytorch works fine,
2 - After getting this error I ensured that the path leads to the correct installation folder of CUDA, I have reisntalled CUDA
3 - I downloaded CUDNN and added it to enviroment variables and PATH with name CUDNN_HOME as a suggestion I found from an older issue regarding the same problem.
4 - I've restarted the comptuer to ensure that new path is indeed loaded,
5 - Used dependency walker on the bin folder of both CUDA and CUDNN, CUDA had a missing dependency for the nvprof.exe which I solved by adding a directory including that library to PATH
6 - I've attempted to Add the CUDNN libraries, binaries and headers directly to the CUDA folder

The conda evironment is made only for this project so its a clean one, so I haven't attempted starting a new one.

To reproduce

from ultralytics import YOLOv10

model = YOLOv10("../models/best_v10n_2.pt", verbose=False)
result = model.val(data="preprocessed_annotations/data.yaml", amp=True)

Outputs :

YOLOv8.1.34 🚀 Python-3.12.4 torch-2.4.0+cu121 CUDA:0 (NVIDIA GeForce RTX 2060, 6144MiB)...

model.export(format='onnx')
model = YOLOv10("models/best_v10n_2.onnx", verbose=False, task='detect')
model.val(data="data/preprocessed_annotations/data.yaml")

Outputs :

*************** EP Error ***************
EP Error [D:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:891](file:///D:/a/_work/1/s/onnxruntime/python/onnxruntime_pybind_state.cc:891) onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page  (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements),  make sure they're in the PATH, and that your GPU is supported.
 when using ['CUDAExecutionProvider', 'CPUExecutionProvider']
Falling back to ['CUDAExecutionProvider', 'CPUExecutionProvider'] and retrying.
****************************************

import os
os.environ

'CUDA_PATH': 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.5',
        'CUDA_PATH_V12_5': 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.5',
        'CUDNN_HOME': 'C:\\Program Files\\NVIDIA\\CUDNN\\v9.2',

path includes

'C:\\Program Files\\NVIDIA\\CUDNN\\v9.2\\bin',
 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.5\\bin',
 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.5\\libnvvp',

Urgency

Yes, I have a project submission that is due next thursday

Platform

Windows

OS Version

10.0.19045 Build 19045

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.18.1

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 12.5

[tianleiwu]

First, make sure onnxruntime-gpu is for cuda 12, If you use pip install onnxruntime-gpu, it will install cuda 11 version. The right one is like

pip install onnxruntime-gpu==1.18.1 --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ --force-reinstall

Then you also need install latest VC Runtime:
https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170

If these do not work, you might try this

[FishGPT]

First, make sure onnxruntime-gpu is for cuda 12, If you use pip install onnxruntime-gpu, it will install cuda 11 version. The right one is like

pip install onnxruntime-gpu==1.18.1 --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ --force-reinstall

Then you also need install latest VC Runtime: https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170

If these do not work, you might try this

Thanks, very helpful :D

Windows 11
CUDA 12.4.0
CUDNN 9.3.0.75
PyTorch 2.4.0

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.