Lazy import cuSOLVER

[kmaehashi]

This is a part of #7620.

I tried to softlink cuSOLVER as I did in NVRTC but found there is a native function calls to cuSOLVER in cupy_lapack.h where softlink doesn't work.
For cuSOLVER I'm thinking of using lazy-import instead of softlink.

Depends on #7840 (merged).

[leofang]

What if I change to use dlopen in cupy_lapack.h? Would it help unify the style?

[kmaehashi]

TBH, after working on this PR, I'm a bit surprised how it is simple to lazy-import cuSOLVER than I initially expected. Although I still think soft linking (or cuSOLVER/cuSPARSE/cuFFT/... support in CUDA Python) is something we will need to support "one CuPy package for all CUDA versions", this is not mandatory to achieve #7620 in v13.

For the modules listed in #7620, I'm currently assessing only Runtime & Driver require soft linking (which is imported/cimported from many places), and the remaining can be done just by lazy-importing. I'm thinking of next try lazy-importing cuSPARSE and see if my assumption is correct.

As Jitify and CUB modules also have C/C++ level dependencies to CTK, we will need lazy-importing for them anyway.

[kmaehashi]

/test mini

[leofang]

For the modules listed in #7620, I'm currently assessing only Runtime & Driver require soft linking (which is imported/cimported from many places), and the remaining can be done just by lazy-importing. I'm thinking of next try lazy-importing cuSPARSE and see if my assumption is correct.

As Jitify and CUB modules also have C/C++ level dependencies to CTK, we will need lazy-importing for them anyway.

In light of #7727, I would suggest to always statically link to cudart and soft-link to driver/NVRTC. Then,

1. this eliminates completely cudart from the discussion
2. this follows CUDA Python's current approach
3. the CUB module no longer has external dependency (cudart is its sole dependency)

[kmaehashi]

In light of #7727, I would suggest to always statically link to cudart and soft-link to driver/NVRTC.
...
3. the CUB module no longer has external dependency (cudart is its sole dependency)

Is it safe to mix multiple versions of CUDA runtimes within the same process? If so, let's static link CUDART to CUB to avoid lazy-importing. 😃

[leofang]

Is it safe to mix multiple versions of CUDA runtimes within the same process?

If you're referring to static linking to cudart, yes it is safe by design. As long as all DSOs see the same user-mode driver (libcuda), which is already upheld by design, it'd work just fine. Think of it as removing cudart from the dependency chain:

any CUDA library -> libcudart -> libcuda

[leofang]

(btw NVCC by default statically links to cudart, I suspect we don't see that for any CuPy modules because we use gcc instead of nvcc to link, which is also fine, just FYI)

[takagi]

LGTM!

[kmaehashi]

Thanks for the clarification, Leo! Just submitted a PR to static link cudart to CUB (#7850)