Vulkan: compute buffers +630mb with 1.100 vs .99?

[EmanuelOverride]

I'm running into "failed to allocate compute pp buffers" with configurations that I've been using with versions 1.99 and below. For running 1.100, I'm having to drop layers and reduce context size to make everything fit. Comparing the two logs, same model and settings, it appears like the requested compute buffer size has grown by, what, 60%?, and I'm running things pretty tight so the failure is expected. Is that increase expected for some reason, and/or am I being stupid?

Welcome to KoboldCpp - Version 1.100...

load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: relocated tensors: 12 of 291

PrefetchVirtualMemory skipped in compatibility mode.
load_tensors: offloading 31 repeating layers to GPU
load_tensors: offloaded 31/33 layers to GPU
load_tensors:      Vulkan0 model buffer size =  3860.00 MiB
load_tensors:   CPU_Mapped model buffer size =  4685.30 MiB

Automatic RoPE Scaling: Using (scale:1.000, base:1776946.1).
llama_init_from_model: model default pooling_type is [0], but [-1] was specified
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 16512
llama_context: n_ctx_per_seq = 16512
llama_context: n_batch       = 512
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = disabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1776946.1
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (16512) > n_ctx_train (8192) -- possible training context overflow
set_abort_callback: call
llama_context:        CPU  output buffer size =     0.49 MiB
create_memory: n_ctx = 16512 (padded)
llama_kv_cache:    Vulkan0 KV buffer size =  1999.50 MiB
llama_kv_cache:        CPU KV buffer size =    64.50 MiB
llama_kv_cache: size = 2064.00 MiB ( 16512 cells,  32 layers,  1/1 seqs), K (f16): 1032.00 MiB, V (f16): 1032.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
llama_context: max_nodes = 2328
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
ggml_vulkan: Device memory allocation of size 1082130432 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
ggml_gallocr_reserve_n: failed to allocate Vulkan0 buffer of size 1784127488
graph_reserve: failed to allocate compute buffers
llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
gpttype_load_model: error: failed to load model

Welcome to KoboldCpp - Version 1.99.4.....

load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: relocated tensors: 12 of 291

PrefetchVirtualMemory skipped in compatibility mode.
load_tensors: offloading 31 repeating layers to GPU
load_tensors: offloaded 31/33 layers to GPU
load_tensors:      Vulkan0 model buffer size =  3860.00 MiB
load_tensors:   CPU_Mapped model buffer size =  4685.30 MiB

Automatic RoPE Scaling: Using (scale:1.000, base:1776946.1).
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 16512
llama_context: n_ctx_per_seq = 16512
llama_context: n_batch       = 512
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = disabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1776946.1
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (16512) > n_ctx_train (8192) -- possible training context overflow
set_abort_callback: call
llama_context:        CPU  output buffer size =     0.49 MiB
create_memory: n_ctx = 16512 (padded)
llama_kv_cache:    Vulkan0 KV buffer size =  1999.50 MiB
llama_kv_cache:        CPU KV buffer size =    64.50 MiB
llama_kv_cache: size = 2064.00 MiB ( 16512 cells,  32 layers,  1/1 seqs), K (f16): 1032.00 MiB, V (f16): 1032.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
llama_context: max_nodes = 2328
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
llama_context:    Vulkan0 compute buffer size =  1153.75 MiB
llama_context: Vulkan_Host compute buffer size =    44.26 MiB
llama_context: graph nodes  = 1126
llama_context: graph splits = 15 (with bs=512), 3 (with bs=1)
Threadpool set to 4 threads and 8 blasthreads...

Power Throttling skipped in compatibility mode.
attach_threadpool: call
Starting model warm up, please wait a moment...
Load Text Model OK: True
Embedded KoboldAI Lite loaded.
Embedded API docs loaded.

EDIT: Version 1.100.1 same thing. Reverting back to koboldcpp_vulkan_noavx2.dll from 99.4 solves it, but I'm guessing that kills the image gen related changes? (Yep, obv.) I haven't made any serious attempt at torturing my old hardware with any of that yet. ;)

[LostRuins]

Can you provide the full untruncated loading logs for both cases?

[LostRuins]

From the existing logs snippets you provided, the requested allocation amounts look exactly the same between those versions.

I think the only thing that has changed regarding vulkan alloc is ggml-org#16354 , I don't know of anything else that would cause a previously fine allocation to stop working on the same setup with the same buffer sizes requested.

@jeffbolznv do you know of any scenario or setup where ggml-org#16354 could actually result in a reduced allocable limit?

[jeffbolznv]

The main part of that change I'd expect might cause an allocation failure is this: https://github.com/ggml-org/llama.cpp/pull/16354/files#diff-35a5049d5eebe22eda1e0d661bd87639b31aafeba62deeaaaca9c13ec3e71d11R2016. What kind of device is this and what does it report for max allocation size and max buffer size?

[EmanuelOverride]

From the existing logs snippets you provided, the requested allocation amounts look exactly the same between those versions.

They do? I was comparing the amount allocated in 1.99.4 (1153.75) with the amount failing to allocate in 1.100.1 (1784127488) and assuming that was the difference, and that's roughly how much I've been cutting. Here are the full logs:

1.100.1.txt
1.99.4.txt

EDIT: And here's a log 1.100.1 with the same config, same model -2 layers:

1.100.1B.txt

Notice: Vulkan0 compute buffer size = 1701.48 MiB

EDIT2: And, for the hell of it, I ran a benchmark on 1.100.1 -2 layers v.s 1.99.4, and turns out it fails, despite initially starting with more vram available:

Processing Prompt [BLAS] (14848 / 16284 tokens)ggml_vulkan: Device memory allocation of size 973078528 failed.

So it's an initial +600 mb, and then an additional inference cost of +300 mb. Tried it with another -2 layers and noticed 1 gig of empty space sitting unused, once or twice filling up to capacity for a split second. Maybe it's a unique situation for me and my ancient hardware, but figured it's worth pointing out in case it's worth something.

[wbruna]

Possibly related: ggml-org#16679

[LostRuins]

I'm not sure what might cause the graph split changes - haven't really looked into all that stuff. Ideally if you wanted to test 29 layers in 1.100.1, you should compare with 29 layers in 1.99.4 too.

Anyway, I've created two new CI builds:

1. with the VK_WHOLE_SIZE changes reverted and with PR 16679: https://github.com/LostRuins/koboldcpp/actions/runs/18751635959
2. with just PR 16679: https://github.com/LostRuins/koboldcpp/actions/runs/18751719744

Builds will be ready in 2 hours from now. let me know if either test build fixes it @EmanuelOverride

[EmanuelOverride]

Cheers. Had a go with the builds:

1. VK_WHOLE_SIZE+pr 16679: No difference from regular 1.100.1, at least that I can see.

VK_WHOLE_SIZE-PR16679.txt

2. Just PR 16679: No difference from regular 1.100.1, at least that I can see.

PR16679.txt

====

I did however also test lowering the context, and from observation it appears that 16k is some sort of breakpoint, because at 14k and below, everything runs exactly like normal (And this is true for regular 1.100/.1 also):

14k1.100.1@33-968.26 MiB: VK_WHOLE_SIZE-PR16679sub16k.txt

14k1.99.4@33-968.26 MiB: 1.99.4sub16k.txt

====

16k1.100.1@29-1701.48 MiB: VK_WHOLE_SIZE-PR16679_16k.txt

16k1.99.4@29-1153.75 MiB: 1.99.4_16k.txt

====

Then, going higher into 20k, 1.100.1 keeps the "bump" it got when crossing into 16k, but actually allocates a little less overall (24mb) to the buffer, than 1.99.4. The increase from 16 to 20k is 256 vs 280 mb respectively:

20k1.100.1@24-1957.48 MiB: VK_WHOLE_SIZE-PR16679_20k.txt

20k1.99.4@24-1433.75 MiB: 1.99.4_20k.txt

====

So, I guess the discrepancy basically solves itself if I just max out the context size at 131k. ;) In all seriousness, it may affect more people than me, but not be particularly noticeable at higher context limits (although at 32k it's still 450mb) and, as evident, not noticeable at all below 16k. Any ideas what could be so special about 16k?

Edit: As for,

I'm not sure what might cause the graph split changes - haven't really looked into all that stuff.

Made me take a closer look at the logs. Graph splits stay the same between the two versions, depending on number of layers being offloaded, so I guess any perceived change is from me presenting imperfect comparisons with differing numbers of offloaded layers. The only difference seems to be in graph nodes, 1158 for 1.100.1 and 1126 for 1.99.4, but I'm guessing that's expected.

[LostRuins]

Sorry then, if reverting had no effect I honestly have no idea, definitely something upstream changed but you'd probably have to open an issue there in order to get it looked at. I wonder if there's something with the 16k threshold, #1770 also mentions issues going above 16k.

[EmanuelOverride]

@LostRuins Just to be absolutely clear, so that there are no misunderstandings, I tested both versions using Vulkan (Old CPU) as the backend, since my rig does not support AVX2. Just making that clear, in case building for Windows vs Windows OldPC adds some sort of wrinkle.

• That said, I did emulate AVX2 with Intel-sde to check, and it made no difference to the observed behavior.

• Also benched 1.96.2-1.97.1-1.99.4 with 16k since Vulkan considerably slower on large context after 1.97.1 #1770 was mentioned. Both earlier versions show comparable speeds within margin of error, while 1.99.4 is actually consistently faster +9-10T/s processing. Having benched 1.99.4 v.s 1.100.1 several times already, that performance increase is retained between versions. Don't see any obvious connection of this issue with 1770, since all 3 earlier versions are consistent in how compute buffers are allocated.

• Tested ClBlast to determine whether it's actually a Vulkan-specific issue, and compute buffers indeed remain comparable between versions, at 1153.75 MiB.

• Tested Cuda with Zluda emulation, for same reason, and compute buffers remain comparable between versions, at 1153.75 MiB. (Zluda currently fails after allocating all the memory, but that's a separate topic.) Okay, so it is indeed a Vulkan-specific issue.

====

Tested llamacpp b6833 directly, with a comparable config. Same basic observable behaviors, except more oddities. I started testing different context sizes, to look for the breakpoint, and it seems to be exactly at 14592-14593.

• 14.592k n-prompt: 1022.50 MiB
• 14.593k n-prompt: 1910.13 MiB

However, I didn't expect that much of an increase, so I mapped another breakpoint, exactly at 15040-15041.

• 15.040k n-prompt: 1963.75 MiB
• 15.041k n-prompt: 1611.48 MiB

I didn't test much further than 20k, but I guess there could be additional jumps at multiples of these values, say 29184. However, I would be particularly curious to know if this issue is specific to my hardware of if it's a common and repeatable phenomenon.

Finally, I traced down the last workable version b6567 so, whatever changed, changed with the release of b6568. Guess I'll open an issue over there, as per your recommendation. Cheers.

[LostRuins]

Thanks!

[EmanuelOverride]

Setting GGML_VK_SUBALLOCATION_BLOCK_SIZE to 2GB (the purported maximum of AMD's proprietary vulkan driver) restores expected behavior and allocates 1053.13 MiB to the compute buffer (in llama-bench). It is, for whatever reason, not a solution for koboldcpp, which errors out (scratch that, non-repeatable), but setting the block size to 256mb instead seemingly works fine, and only accrues a small 20mb overhead over the nominal value. On the face of it, this seems like a reasonable intermediary solution in my specific case.

Although the jury is still out on what causes such a large overhead, the change responsible is that the compute buffer is now getting chunked, with a default breakpoint at 1024 MB. I was doing some quick tests with both llama and kobold as I was composing my reply, and regretfully did not preserve the error originally thrown by kobold when switching the block size to 2gb, and on subsequent attempts have not been able to reproduce it.

With PR 16679, I've done a handful of test runs and haven't had any issues at all, and see another performance increase when benching. Not sure if it's placebo at this point, or if I'm imagining things. Should I close this as resolved?

To my mind, there's still the question of whether it's in some way related to 16679 and whether the observed behavior occurs for others as well -- raising the block-size recreates the previous configuration of a single in practical terms "infinite" block, to circumvent the breakpoint bloat entirely, while lowering the block-size cuts the bloat and spreads it over multiple breakpoints, making it negligible, but a comprehensive solution is probably above my paygrade. I'm just happy to have something that works, so thanks a lot.

[LostRuins]

Yeah I think we can close it