Huge Slowdown

[Oruli]

So, I think this aligns with Comfy 3.15 update but cannot be 100% sure. But a low res inference is taking me 1m35s roughly with this node and GGUF model, but BF16 base model is taking only 54s.

[pollockjj]

@Oruli,

I'm sorry to hear you are having a slowdown. In order for me to help, I will need to know a bit more about the node you're referring to as it could be one of two:

1. Are you talking about the standard MultiGPU GGUF Loader, or
2. Are you talking about the Virtual VRAM loader?

If Virtual VRAM, what were your settings? If you would be so kind, please collect data at these steps:

1. What is the generation time if you just use the standard ComfyUI-GGUF loader?
2. What is the generation time if you set Virtual VRAM to zero?
3. What is the generation time if you set Virtual VRAM to maximum?

I know City96 pushed out an update to help out the 50XX folks, so that might be related as well. If you haven't I might check there to see if others are seeing it while I start checking over here.

Once you get back to me with that data (including anything else like OS, DRAM, etc., Fresh reboot, etc., # of LoRAs used, if any etc.) we can start narrowing-down what might be the cause the slowdown you are seeing. Unless you added a couple of LoRAs in there and this is a simple stacking issue, that additional data will help. The good news is that I haven't pushed anything out for a bit, so it is unlikely to be core-MultiGPU unless it is interacting with City's latest code (which was in an unrelated part to the area I patch.)

[Oruli]

1. Are you talking about the standard MultiGPU GGUF Loader, or
2. Are you talking about the Virtual VRAM loader?

I am using the nodes UnetLoaderGGUFDisTorchMultiGPU, VAELoaderMultiGPU and DualCLIPLoaderGGUFMultiGPU

Prior to this I have never used GGUF models always the safetensors. The interference with your nodes and the GGUF was the same time, but enabled me to push the resolution limits. However now as mentioned it's so much slower.

List of results from different models/loading:

BF16 default - 0:49
BF16 fp8_e4m3fn - 0:40
GGUF - 1:07
GGUF (your node) - 1:09

note: I am using Wavespeed's "apply first block cache" and sageattention

If Virtual VRAM, what were your settings? If you would be so kind, please collect data at these steps:

I'm using 8GB virtual VRAM, I'm also using --reserve-vram 4gb

1:09

1. What is the generation time if you just use the standard ComfyUI-GGUF loader?

1:07

2. What is the generation time if you set Virtual VRAM to zero?

1:01

3. What is the generation time if you set Virtual VRAM to maximum?

what is maximum?

Once you get back to me with that data (including anything else like OS, DRAM, etc., Fresh reboot, etc., # of LoRAs used, if any etc.) we can start narrowing-down what might be the cause the slowdown you are seeing. Unless you added a couple of LoRAs in there and this is a simple stacking issue, that additional data will help. The good news is that I haven't pushed anything out for a bit, so it is unlikely to be core-MultiGPU unless it is interacting with City's latest code (which was in an unrelated part to the area I patch.)

64GB system RAM, Linux.

Loras with GGUF

0 lora - 0:44
1 lora - 0:50
2 lora - 0:57
3 lora - 1:07

I note that when using normal BF16 model lora doesn't change the inference time at all, not even by a second.

[pollockjj]

This is known as the "GGUF LoRA stacking penalty".

You can avoid this using the FP16 .gguf of the safetensor you are have switched from. It is functionally identical to the full .safetensor, and, because the layers do not need to be dequantized, it does not suffer from the same effect.

May I ask which model?

[Oruli]

This is known as the "GGUF LoRA stacking penalty".

You can avoid this using the FP16 .gguf of the safetensor you are have switched from. It is functionally identical to the full .safetensor, and, because the layers do not need to be dequantized, it does not suffer from the same effect.

May I ask which model?

I'm using Hunyuan, it needs loras so I'm surprised nobody has mentioned this already. Though I could swear it was faster than ST models a week ago, not slower.

Doesn't using the F16 gguf defeat the point of trying to lower vram usage though?

I am testing with hunyuan-video-t2v-720p-Q8_0.gguf which is 14GB, the BF16 is 25GB...

[pollockjj]

Hey, @Oruli,

Doesn't using the F16 gguf defeat the point of trying to lower vram usage though?

Yes, it does, to a significant extent in some cases depending on hardware. There are tradeoffs here, and I wasn't offering you an easy fix, just a potential path. In your case, with 64G of memory and linux, there is a good chance that you could use that FP16 GGUF model, set your Virtual VRAM to 24 (it automatically offloads anything greater than 24 so you will get the full model offloaded) and successfully pull those dequantized layers faster from DRAM than the system currently can pull the smaller, quantized layers across the PCIe bus, dequantize, then apply each of the LoRAs you are using serially to those layers.

It needs loras so I'm surprised nobody has mentioned this already.

So, the good news is that this penalty is additive, not multiplicative. Given that generally people are maximizing the Pixel Load of their cards it is seen less than say, Flux, where it is a major issue. As that load gets bigger, the percentage that one second / iteration takes up becomes smaller and smaller.

If you are familiar with nvtop, I would love to see an nvtop run during an FP16 and Q8_0 run if you are so inclined to help collect some data for me.

(And there are other ways to work around this but none of them are one-click and depend on the stability of your LoRA stack. City96's code has proven to be hard to make more optimized than it already is for a problem that only stable diffusion GGUFs suffer from, so it is a small community from where any ongoing optimization passes are coming from.)

[Oruli]

Is the FP16 model the BF16 I'm talking about here? https://huggingface.co/city96/HunyuanVideo-gguf/tree/main

Yes I use NVTOP, will need to figure out how to monitor a whole session run though

note: If i set VRAM to 24 my system swaps out

[pollockjj]

Yes,

https://huggingface.co/city96/HunyuanVideo-gguf/resolve/main/hunyuan-video-t2v-720p-BF16.gguf

That might be faster for you.

[pollockjj]

I imagine you might have more load on that linux system than just Comfy. If so, it is really about how much free memory you have of that 64G that you can assign to the 25G of the model. The remainder will need to reside in VRAM.

[Oruli]

I imagine you might have more load on that linux system than just Comfy. If so, it is really about how much free memory you have of that 64G that you can assign to the 25G of the model. The remainder will need to reside in VRAM.

Nope I have pretty much everything shut down when running, it seems though after a run or two comfy holds almost all of my RAM.

I've just ran the BF16, it takes 1:16 v/s 0:50 and it takes all 24GB of VRAM with 8GB of "virtual VRAM" selected

This compares to around 15GB with the Q8. I'm assuming 2nd pass will OOM.

[pollockjj]

Are you running --lowvram? With that system you shouldn't be accumulating on the CPU/DRAM between runs.

Size of your video card?

[Oruli]

I'm only running "--reserve-vram 4 --use-sage-attention"

It's a 24GB 3090

[pollockjj]

If you're open to it, switch to simply "--lowvram" or as close as you can to that and let me know how it goes.

I have a 2x3090 96G linux system and that is how I run (usually because I am messing with the full .safetensors as part of my testing, but I get good results regardless)

Let me know how your generation times compare to my own data from my own personal rig, if you are so inclined! (These are 0 LoRAs but they get totally washed out as you get to full 24G pixel loads)

Cheers!

[Oruli]

If you're open to it, switch to simply "--lowvram" or as close as you can to that and let me know how it goes.

What do you mean by this, as close as I can get to it? How are you logging and charting data like that, I would be very interested to know exactly how my tests are performing.

I have 2 x 5090 coming in a few weeks.