Add sm_120 (Blackwell / RTX 50-series) support to CUDA 12 backend

[NORDIT]

Summary

The LLamaSharp.Backend.Cuda12 NuGet package does not include CUDA compute capability sm_120 (NVIDIA Blackwell architecture — RTX 5080, RTX 5090, etc.). This means the native llama.dll / libllama.so binaries cannot run on Blackwell GPUs, silently falling back to CPU.

Current Behavior

On an RTX 5080 (Blackwell) with CUDA 12 drivers:

• LLamaSharp.Backend.Cuda12 0.25.0 / 0.26.0 loads but falls back to CPU inference
• No error is raised — the model just runs on CPU instead of GPU
• Users have no indication that their GPU is unsupported by the compiled binaries

Expected Behavior

The CUDA 12 backend should include sm_120 in the CMake build targets so that Blackwell GPUs are utilized for inference.

Workaround

We are currently building llama.cpp from source with -DGGML_CUDA_ARCHITECTURES=120 and packaging the resulting native binaries as a custom NuGet package (LLamaSharp.Backend.Cuda12.Blackwell). This works but requires version-locked alignment with the managed LLamaSharp package (e.g., native 0.25.0 + managed 0.26.0 = ABI mismatch and garbage output).

Suggested Fix

Add sm_120 (and ideally sm_120a) to the CUDA architecture list in the CI build for LLamaSharp.Backend.Cuda12. This is typically a one-line change in the CMake configuration:

set(GGML_CUDA_ARCHITECTURES "60;70;75;80;86;89;90;120")

Or if using the llama.cpp CMake flag:

-DGGML_CUDA_ARCHITECTURES="60;70;75;80;86;89;90;120"

Environment

• GPU: NVIDIA RTX 5080 (Blackwell, sm_120)
• CUDA Driver: 13.1
• CUDA Toolkit: 12.9
• OS: Windows Server 2025 / also need Linux support
• LLamaSharp: 0.25.0 and 0.26.0 tested
• .NET: 10.0

[aropb]

I'm using a 5090 and everything works with CUDA 12.9

[NORDIT]

@aropb Thanks for the data point! Could you share a bit more about your setup?

• Which LLamaSharp / backend NuGet package version are you using?
• Are you using the prebuilt LLamaSharp.Backend.Cuda12 NuGet, or did you compile llama.cpp from source with sm_120?

We're on CUDA toolkit 13.1 (driver 591.74) with an RTX 5080 and tested both LLamaSharp 0.25.0 and 0.26.0 — the prebuilt CUDA 12 backend falls back to CPU silently because the bundled .so/.dll wasn't compiled with Blackwell compute capability. If you're running against the prebuilt backend and it's working on your 5090, that would suggest a newer release may have added sm_120 support.

[aropb]

I am currently using llamasharp 0.26.0, the nuget package is the same, I take the binary files from llamasharp binaries https://github.com/SciSharp/LLamaSharpBinaries/releases/tag/506bb6e01009058f355

It is important to use cuda toolkit 12

[whitebehemoth]

Another workaround (found by Claude Opus) - pre-loading from sdk-12 bunch of .so files in this order.

• libggml-base.so
• libggml-cpu.so
• libggml-cuda.so
• libggml.so

Thus, on backend load linker has them.

for example

var ggmlLib = Path.Combine(cuda12Dir, "libggml.so");
if (File.Exists(ggmlLib)) dlopen(ggmlLib, RTLD_NOW | RTLD_GLOBAL);
...
    [DllImport("libdl.so.2", EntryPoint = "dlopen")]
    private static extern IntPtr dlopen(string filename, int flags);

Environment

• GPU: NVIDIA RTX 5060 ti (Blackwell, sm_120)
• CUDA Driver: 13.1
• CUDA Toolkits: 12.9 and 13.1 (installed first)
• OS: Ubuntu 24.04
• LLamaSharp: 0.26.0
• .NET: 10.0

[m0nsky]

@aropb Thanks for the data point! Could you share a bit more about your setup?

• Which LLamaSharp / backend NuGet package version are you using?
• Are you using the prebuilt LLamaSharp.Backend.Cuda12 NuGet, or did you compile llama.cpp from source with sm_120?

We're on CUDA toolkit 13.1 (driver 591.74) with an RTX 5080 and tested both LLamaSharp 0.25.0 and 0.26.0 — the prebuilt CUDA 12 backend falls back to CPU silently because the bundled .so/.dll wasn't compiled with Blackwell compute capability. If you're running against the prebuilt backend and it's working on your 5090, that would suggest a newer release may have added sm_120 support.

I've ran into this issue before, you need CUDA 12.x

[zsogitbe]

I’d like to clarify a common misconception regarding the CUDA 12 backend and Blackwell support: the assumption that the backend falls back to CPU specifically because the bundled .so/.dll wasn't compiled for Blackwell (sm_120) is actually incorrect.

CUDA is built with backward compatibility at its core. If a binary is compiled for an older compute capability (e.g., sm_80 or sm_90), it will still run on a Blackwell GPU. While it might be slightly slower than a version natively optimized for sm_120, it won't simply 'fail' or trigger a CPU fallback for that reason alone.

There are two practical reasons why every architecture isn't bundled into the default release:

1. Binary Size: Every compute capability added to the build significantly increases the size of the .dll or .so. To keep the package sizes reasonable, maintainers must select a few common, high-impact targets.
2. Maintenance: Targeting brand-new architectures often requires the very latest CUDA Toolkits, which may introduce other compatibility issues for users on older drivers.

If people are seeing a silent fallback to CPU, the cause is more likely a driver version mismatch, a missing dependency, or a specific runtime initialization error (which can sometimes happen with new hardware architectures in early driver releases).

For anyone who wants the absolute peak performance for their Blackwell card, the beauty of LLamaSharp/llama.cpp is that you can always recompile the backend from source with your specific compute capability (e.g., -DCMAKE_CUDA_ARCHITECTURES=120) to get a natively optimized binary.