Skip to content

Commit 3f9fe3e

Browse files
dhiltgenrick-github
authored andcommitted
cuda: leverage JIT for smaller footprint (ollama#11635)
Prior to this change our official binaries contained both JIT PTX code and the cubin binary code for our chosen compute capabilities. This change switches to only compile the PTX code and rely on JIT at runtime for generating the cubin specific to the users GPU. The cubins are cached on the users system, so they should only see a small lag on the very first model load for a given Ollama release. This also adds the first generation of Blackwell GPUs so they aren't reliant on the Hopper PTX. This change reduces the ggml-cuda.dll from 1.2G to 460M
1 parent 947c24f commit 3f9fe3e

File tree

1 file changed

+3
-3
lines changed

1 file changed

+3
-3
lines changed

CMakePresets.json

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,22 +22,22 @@
2222
"name": "CUDA 12",
2323
"inherits": [ "CUDA" ],
2424
"cacheVariables": {
25-
"CMAKE_CUDA_ARCHITECTURES": "50;60;61;70;75;80;86;87;89;90;90a;120",
25+
"CMAKE_CUDA_ARCHITECTURES": "50-virtual;60-virtual;61-virtual;70-virtual;75-virtual;80-virtual;86-virtual;89-virtual;90-virtual;90a-virtual;100-virtual;120-virtual",
2626
"CMAKE_CUDA_FLAGS": "-Wno-deprecated-gpu-targets -t 2"
2727
}
2828
},
2929
{
3030
"name": "JetPack 5",
3131
"inherits": [ "CUDA" ],
3232
"cacheVariables": {
33-
"CMAKE_CUDA_ARCHITECTURES": "72;87"
33+
"CMAKE_CUDA_ARCHITECTURES": "72-virtual;87-virtual"
3434
}
3535
},
3636
{
3737
"name": "JetPack 6",
3838
"inherits": [ "CUDA" ],
3939
"cacheVariables": {
40-
"CMAKE_CUDA_ARCHITECTURES": "87"
40+
"CMAKE_CUDA_ARCHITECTURES": "87-virtual"
4141
}
4242
},
4343
{

0 commit comments

Comments
 (0)