[CUDA:Bugfix] Fall back to PTX for GPUs newer than the toolkit

[Qxinyu]

Description

Reported in #4883: on WSL with an RTX 5060 and CUDA 12.4, pymnn/pip_package/build_deps.py llm,cuda either fails at configure or produces a library that never touches the GPU, and inference then drowns in bogus CUDA alloc failed / Out of memory when gamma is acquired in CudaLayerNorm lines.

The root cause is in SelectCudaComputeArch.cmake. CUDA_LIMIT_GPU_ARCHITECTURE — the "newest arch this toolkit can target" clamp consumed by CUDA_DETECT_INSTALLED_GPUS — is only ever set for toolkits older than 12.0. On CUDA 12.0-12.7 a Blackwell GPU is therefore passed straight through:

• sm_120 (RTX 50xx) -> -gencode arch=compute_120, which nvcc 12.4 rejects outright.
• sm_100 (B100/B200) -> compute_100, also rejected, but only after support_archs fires Please add your own sm arch 100 to CmakeLists.txt! — advice that, if followed, yields a binary that still cannot load.

Neither path emits PTX, so there is nothing for the driver to JIT.

Changes

1. Set the clamp to 10.0 for every toolkit below 12.8 (Hopper is the newest those can target natively). A too-new GPU now degrades to 9.0+PTX and the driver JITs it.
2. Append +PTX explicitly in the clamp path. It previously relied on the last entry of CUDA_COMMON_GPU_ARCHITECTURES already carrying +PTX, which is not true for CUDA 12.0+.
3. Emit a CMake WARNING naming the detected capability, the toolkit version and the fallback, and noting that prebuilt vendor libraries may still refuse to initialize. On Blackwell + CUDA 12.4 cublasCreate returns CUBLAS_STATUS_NOT_INITIALIZED no matter what MNN compiles, so the warning says the toolkit upgrade is still required rather than letting the user discover it as 500 lines of fake OOM.
4. Delete set(CUDA_LIMIT_GPU_ARCHITECUTRE "8.6") — misspelled, therefore dead. Correcting the spelling instead would be a regression: it would set the clamp to 8.6 on CUDA >= 12.0 and downgrade sm_86 / sm_89 GPUs to sm_90.
5. Move the CUDA 11.8 clamp from 9.0 to 10.0 so a Hopper GPU on CUDA 11.8 keeps native sm_90 SASS instead of being downgraded to the PTX fallback.
6. Reword the support_archs FATAL_ERROR to state that editing the list does not help, that the fused-attention plugins only ship cubins for sm_75/80/86/89, and to point at -DCUDA_ARCHS=<major.minor>.

Verification

No CUDA toolchain is available locally, so this is verified at the CMake level: CUDA_SELECT_NVCC_ARCH_FLAGS is driven directly with cmake -P over CUDA {10.2, 11.0, 11.4, 11.8, 12.4, 12.8} x detected capability {7.0, 7.5, 8.0, 8.6, 8.9, 9.0, 10.0, 12.0}, comparing the complete generated nvcc flags before and after.

12 of the 15 reachable combinations are byte-identical. The 3 that differ are exactly the intended ones:

toolkit	GPU	before	after
12.4	sm_120	compute_120 (nvcc rejects)	sm_90 + compute_90 PTX + warning
12.4	sm_100	compute_100 (nvcc rejects)	sm_90 + compute_90 PTX + warning
11.8	sm_90	sm_90 + redundant PTX	sm_90

A real nvcc build on Hopper / Blackwell hardware is still needed before merging.

Not addressed here

Adding 90 to support_archs looks like the obvious companion fix but is not safe on its own, so it is deliberately left out:

• MNN_CUDA_NATIVE_ARCH=ON (which build_deps.py always passes for CUDA) defines MNN_CUDA_ENABLE_SM90, and fmha_flash_attention.h / fmha_cross_attention.h hard-#error unless one of SM75/80/86/89 is defined. Configure would pass and the build would fail later.
• With MNN_CUDA_NATIVE_ARCH=OFF the hardcoded gencode list never emits sm_90 and emits no PTX, so configure would pass and produce a binary that cannot load on Hopper — turning a loud failure into a silent one.

Supporting Hopper / Blackwell end-to-end needs the fused-attention cubin gap closed first, either by excluding FmhaV2 / Fmhca / FmhaCommon when the target arch has no cubin, or by generating sm_90 / sm_120 cubins. Related latent bug on that path: sharedCubinLoader.h:156-161 logs "Could not find kernel" and then dereferences the end() iterator unconditionally.

Module

CUDA

Type

• Feature
• Bugfix
• Perf
• Refact
• Style
• Doc
• Test
• Chore

Checklist

• Commit message follows [Module:Type] Description format
• Code compiles without errors — no CUDA toolchain locally; CMake logic verified via cmake -P, nvcc build not run
• Tested on relevant platform(s) — needs a Hopper / Blackwell machine
• No unrelated format or style changes included

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[wangzhaode]

Thanks for improving the CUDA architecture detection and diagnostics!

One blocking issue remains in the normal build path: with CUDA 12.4 and detected capability 12.0, the selector now produces sm_90 plus compute_90 PTX, but source/backend/cuda/CMakeLists.txt still rejects 90 in support_archs. We reproduced this with the current selector and support list, so the fallback does not yet get through configuration.

Could you please make the fallback consistent with the downstream architecture validation, emitted code and fused-attention kernel selection? As your description notes, simply adding 90 to the list is not enough while the native-arch path still selects unsupported fused-attention cubins. If that support is outside this PR's scope, a diagnostics-only change that clearly rejects the unsupported configuration would also be reasonable, rather than presenting the fallback as usable.

A focused configure/build check for the intended combination is enough; no broad benchmark is required. The CLA check is also still pending, so please complete that before merging. Thanks!