LoRA Still Influencing Output Despite Setting "scale" to 0

[Linaqruf]

Hi, thanks and great job for implementing Kohya LoRA format into diffusers. However, I'm running into an issue where the LoRA settings don't seem to be working as described in the documentation.

According to the official documentation, setting the scale value in cross_attention_kwargs = {'scale' : 0} should completely nullify the effect of LoRA. The model should rely only on its base weights.

💡 A scale value of 0 is the same as not using your LoRA weights and you’re only using the base model weights, and a scale value of 1 means you’re only using the fully finetuned LoRA weights. Values between 0 and 1 interpolates between the two weights.

So I thought this is interpolation weight merging, similar to Kohya, Auto1111 or ComfyUI implementation. However In my case, even after setting cross_attention_kwargs = {'scale' : 0}, I've noticed that LoRA still appears to influence the results. This seems contradictory to what the documentation states.

Description	Image
Scale 1
Scale 0.5
Scale 0
No Lora

Is this a known issue, or am I possibly misinterpreting how to set up the feature correctly? I'd appreciate any guidance or clarification.

I wrote a notebook to reproduce this issue here
https://colab.research.google.com/drive/1P_PVUoyrPzb15EK9G1FFZIH3YnpwnPaC

[Linaqruf]

Hi @sayakpaul

[sayakpaul]

Thanks for reporting this. The issue is very well described. Will look into it :-)

[sayakpaul]

We are doing two major refactors just as an FYI which might influence this issue:

#4473
#4765

[sayakpaul]

I think I understand what's happening here. Let me try to elaborate in a lucid manner.

We support LoRA attention processors for the attention modules only at the moment. In those processors, you would find a scale argument in their __call__() methods. Here's an example:

diffusers/src/diffusers/models/attention_processor.py

Line 573 in 470d51c

query = attn.to_q(hidden_states) + scale * self.to_q_lora(hidden_states)

However, non diffusers LoRAs might also contain additional modules concerning different network blocks that don't necessarily correspond to attention always. Such as: conv, mlp, etc.

To support those, we rely on: LoRACompatibleLinear and LoRACompatibleConv.

diffusers/src/diffusers/models/lora.py

Line 101 in 470d51c

class LoRACompatibleLinear(nn.Linear):

diffusers/src/diffusers/models/lora.py

Line 80 in 470d51c

class LoRACompatibleConv(nn.Conv2d):

Their call arguments don't have scale argument. This is why even when we specify the scale argument via cross_attention_kwargs the non-attention LoRA modules still remain effective. This will very likely be fixed in #4765.

I hope this elaboration helps to understand what is happening here.

Cc: @patrickvonplaten @isidentical @apolinario

[Linaqruf]

I see, that makes sense. I thought scale was the same as multiplier in their implementation.

https://github.com/kohya-ss/sd-scripts/blob/sdxl/networks/lora_diffusers.py#L156-L172

Looking forward to the fix!

[patrickvonplaten]

That's exactly right @sayakpaul - only that it will probably not directly be fixed in #4765, but we can open a new PR to look into how to fix it :-)

[sayakpaul]

@Linaqruf #4842 (comment)

[Linaqruf]

Thanks @sayakpaul, @patrickvonplaten, and sorry; I totally forgot about this issue. I was doing other stuff last week. 😅

This is great; not only is the scale: 0 problem fixed, but the implementation is also 1:1 with the Kohya LoRA loader. I compared both implementations and the result is about the same.

Scale	Diffusers	Kohya
1
0.5
0
None

I wrote new notebook to reproduce this result

1. With Diffusers 0.21.0.dev = https://colab.research.google.com/drive/1gasmUzOSTrMcwhPngNCtlSMa2fTkyyOE
2. With Kohya LoRA Loader = https://colab.research.google.com/drive/1siic-5ylI5Zu_ROrE-cLPUM_l1SENLZZ