Multi GPU with Vulkan out of memory issue.

[lastrosade]

Running llama.cpp #5832 (9731134)

I'm trying to load a model on two GPUs with Vulkan.

My GPUs have 20 and 11 gigs of VRAM

Loading a Q6_K quant of size 26.27 GiB (6.56 BPW) with -ts "20,11" -c 512 yields:

ggml ctx size =    0.62 MiB
offloading 60 repeating layers to GPU
offloading non-repeating layers to GPU
offloaded 61/61 layers to GPU
   Vulkan0 buffer size = 17458.44 MiB
   Vulkan1 buffer size =  9088.14 MiB
       CPU buffer size =   358.90 MiB

Vulkan0 KV buffer size =    80.00 MiB
Vulkan1 KV buffer size =    40.00 MiB

KV self size  =  120.00 MiB, K (f16):   60.00 MiB, V (f16):   60.00 MiB
Vulkan_Host input buffer size   =    16.01 MiB
   Vulkan0 compute buffer size =   113.00 MiB
   Vulkan1 compute buffer size =   139.00 MiB
Vulkan_Host compute buffer size =    14.00 MiB

ggml_vulkan: Device memory allocation of size 120422400 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory

The math doesn't seem to add up.

A Q5_K_M quant at 22.65 GiB (5.66 BPW) works perfectly fine until I increase the context to 4096.

This can't possibly be context, right? When using HIP on smaller models, I have to push it much harder to OOM, I should be fine with 31GB of VRAM.
Any idea why this happens?

[0cc4m]

Does GPU0 also run a desktop? It might be an issue with vram fragmentation then, maybe. Can you upload the output of vulkaninfo?

I guess the error message about OOM should also contain the device that failed.

[lastrosade]

Both GPUs have monitors plugged into them.

vulkaninfo.txt

[0cc4m]

That's gonna be a little harder to debug, since we don't even know which GPU is running out of memory. If you know a little bit about C++ we could add the GPU to the OOM error message and get that information.

Apart from that you can also run the program with validation layers enabled by building with LLAMA_VULKAN_VALIDATE=1 and see if that outputs any validation issues before the OOM (but I think that's unlikely to be the problem). You can also build with LLAMA_VULKAN_DEBUG=1 which will put everything the Vulkan part of the program does into your console. This is extremely verbose and you should probably just pipe it into a file and upload it for me to take a look at.

Let me know if you want me to provide the patch for the OOM error message or more info about the other steps.

[lastrosade]

Thank you for taking time to help me.

Here's the output from a run with LLAMA_VULKAN_DEBUG=1
dbg.txt
And one with both LLAMA_VULKAN_DEBUG=1 and LLAMA_VULKAN_VALIDATE=1
dbg.txt
The model is a 8x7b of size 23.6 GB.

It fails to allocate 1.3 gigs of VRAM when I can easily use cublas on my 1080ti to fill it to about 11.8 gigs or HIP on my 7900XT up to 20 gigs without issues.
Or is that additive? idk.

As for the custom error message, I have no idea how I would do that.

[0cc4m]

Thank you. As a sidenote, Mixtral is not yet supported on Vulkan, but in this case it's crashing before that would be a problem.

Something is not adding up with these allocation sizes. In your first q5_k_m example it manages to allocate the two compute buffers of size 113 and 139 MB, but fails a later allocation.

In your debug outputs it tries to allocate a 1.3GB compute buffer, which is way too large. In my attempts to reproduce it it only tried to allocate around 320MB per GPU. Something is going wrong there.

But I also see that's an IQ4 quant, and Imatrix quants are also not yet supported on Vulkan. Can you reproduce the issue with a combination of model and quant that's supported? For example, I think a Yi-34B q5k or q6k should be around this size and is supported.

[lastrosade]

Running Dolphin 2.2 Yi 34b 200k

I ran one that worked, just in case this could be useful.

Worked: Q4_K_S, (19,598,649,632 bytes), dbg_Q4_K_S.txt.gz (The text file is 232MB :) )

Q5_K_S, (23,707,691,296 bytes) dbg_Q5_K_S.txt
Q6_K, (28,213,926,176 bytes) dbg_Q6_K.txt

I have done these tests with as little other processes running to try and limit VRAM usage from whatever may be using HW acceleration or whatever.

[0cc4m]

Apologies for not getting back to you sooner, I was too busy last week. Your logs show that the size of the the dequant buffer is the problem here. Because I didn't have proper matmul dequant shaders for the k-quants yet (and also didn't update the buffer size logic yet) they use quite a bit of vram. Too much for your setup, with q5_k and q6_k.

Good news is that I have now implemented the k-quant matmul shaders and will update the buffer size logic to take this into account. That should save you a few hundred megabytes of VRAM and hopefully solve this issue. I'll let you know when you can test this.

[0cc4m]

@lastrosade Please check if #6155 fixes your problem.

[lastrosade]

Apologies for not getting back to you sooner

No biggie.

What (I think) should be working still does not work, same model as before,

Q6_K, (28,213,926,176 bytes) dbg_Q6_K.txt

[0cc4m]

Apologies for not getting back to you sooner

No biggie.

What (I think) should be working still does not work, same model as before,

Q6_K, (28,213,926,176 bytes) dbg_Q6_K.txt

That is still the old code. The new code even had an issue that would have prevented you from building it. I fixed that now.

The PR is not merged yet, do you know how to check out the feature branch or do you need help with that?

[lastrosade]

Sorry, didn't see that it wasn't upstream.

So I built it the same way as before by including -DLLAMA_VULKAN_DEBUG=1 -DLLAMA_VULKAN_VALIDATE=1
This time from branch 0cc4m/vulkan-improvements

Q4_K_S worked as it did before.
Q5_K_S now works! dbg_Q5_K_S.txt (I prematurely stopped it mid-generation.)
Q6_K does not work: dbg_Q6_K.txt
I should definitely have enough memory left for Q6_K to work, but I can't even really tell how much space 4K of context is really taking.