[Core] LoRA improvements pt. 3 #4842

sayakpaul · 2023-08-30T07:17:12Z

What does this PR do?

Addresses Fuse loras #4473 (comment).
Will potentially also fix LoRA Still Influencing Output Despite Setting "scale" to 0 #4751.

Currently, using different scales during LoRA fusion is supported but how to best support it for non-fusion workflows is yet to be brainstormed.

From here:

Make sure that "lora_scale" is applied to all LoRA layers. Instead of passing a cross_attention_kwargs through the attention processors, it's probably better to work with a "setter"-method here that iterates through all layers and if the layer is a LoRA layer it just sets the "lora_scale" for the specific class

Do we do this when a pipeline call is made and if a LoRACompatible* layer has its lora_layer set, then we multiply the scale with the output of the lora_layer. The scale I think still needs to be retrieved from cross_attention_kwargs.

Or, do we set lora_scale attribute in the LoRALinear and LoRAConv layers when they're initialized? But doing so will not allow having different scale values during inference IIUC.

HuggingFaceDocBuilderDev · 2023-08-30T07:26:06Z

The documentation is not available anymore as the PR was closed or merged.

apolinario · 2023-08-30T07:39:03Z

But doing so will not allow having different scale values during inference IIUC.

I think that is not ideal - experimentation with the pipe loaded and changing the LoRA weight on inference time to find the sweet spot is a common use-case in the community, many LoRA authors suggest a range and users usually try to find the best scale within the range for their prompt

Also, slider LoRAs are a growing use-case, and they require different values during inference, as it is expected from users to play with different scale values during inference to see how the scale influence the results

E.g.: the curly hair slider LoRA will build subjects with a super straight hair if the LoRA scale is -8 and super curly if the LoRA scale is 8

Such LoRAs have more than 250K downloads on CivitAI in total, and this technique is growing, so I would consider this use-case seriously in the design of this API

sayakpaul · 2023-08-30T09:55:32Z

Do we do this when a pipeline call is made and if a LoRACompatible* layer has its lora_layer set, then we multiply the scale with the output of the lora_layer. The scale I think still needs to be retrieved from cross_attention_kwargs.

Let me proceed with this one and see how this factors out.

sayakpaul · 2023-08-30T13:14:58Z

I am still working on it. Please don't commit anything to it yet.

src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py

patrickvonplaten

Super nice! Left mostly nits.

It would be nice to call _adjust_lora_scale_text_encoder part directly in the encode_prompt function
In addition to the tests that have been added, can we also test that: fuse lora with scales gives the same result as normal lora with scales?

src/diffusers/pipelines/versatile_diffusion/modeling_text_unet.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

patrickvonplaten

Very nice! Good to merge for me whenever :-)

sayakpaul · 2023-09-04T07:36:21Z

@patrickvonplaten still some problems (specifically the test case you mentioned here) and I am debugging them currently. About the tests failing I think they're acting flaky because without the lora_scale related changes, they sometimes fail and sometimes don't (cc: @DN6).

I will request for another review from you when this PR's ready. Thanks for your reviews so far!

sayakpaul · 2023-09-04T09:16:41Z

@patrickvonplaten in e60f450, I added a test to check for the equivalence (as mentioned in #4842). However, it currently fails. I looked into the potential suspects but they seem alright to me (indicated by the print statements one can see). If you have some time, it would be great to have some suggestions here.

To clear the air, I am shifting to #4855.

sayakpaul · 2023-09-04T11:33:15Z

Back at it.

sayakpaul · 2023-09-04T12:17:16Z

There's something wrong with our LoRA attention processor, likely (maybe due to the changes introduced in #4833).

I am confirming this because if we load non-diffusers LoRA checkpoints the effects of scale on both LoRA fusion and non-fusion seem to be preserved. Here's my exploration notebook that confirms this behavior: https://gist.github.com/sayakpaul/ddeed50516ecbd5be89e760c830da46f.

So, this probably narrows down the area we need to look for to solve the current bug. Looking into it.

patrickvonplaten · 2023-09-04T21:53:10Z

tests/models/test_lora_layers.py

    with torch.no_grad():
        for parameter in lora_attn_parameters:
            if randn_weight:
-                parameter[:] = torch.randn_like(parameter)
+                parameter[:] = torch.randn_like(parameter) * var


We should allow to have smaller LoRA weights so that they we can keep a reasonable precision in the tests.

patrickvonplaten · 2023-09-04T21:54:32Z

@sayakpaul I think your PR was just fine. I'm quite certain that the test only failed due to numerical precision issues which were fixed by passing LoRA layers that are less extreme in values (e.g. are distributed via N(0, 0.1) instead of N(0, 1)).

Also ran the slow tests locally and they all passed. Think we're good here. Nice job!

xhinker · 2023-09-23T18:54:43Z

@sayakpaul Thank you for the PR, I was confused on why you add the

self.lora_scale = lora_scale

in the end of _fuse_lora function, after reading through #4473 , I realized that this line code is for enabling a correct unfuse, if not adding it, the unfuse don't have the scale value to remove the LoRA weight off the base model. am I right?

sayakpaul · 2023-09-23T21:34:06Z

Precisely.

* throw warning when more than one lora is attempted to be fused. * introduce support of lora scale during fusion. * change test name * changes * change to _lora_scale * lora_scale to call whenever applicable. * debugging * lora_scale additional. * cross_attention_kwargs * lora_scale -> scale. * lora_scale fix * lora_scale in patched projection. * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * styling. * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * remove unneeded prints. * remove unneeded prints. * assign cross_attention_kwargs. * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * clean up. * refactor scale retrieval logic a bit. * fix nonetypw * fix: tests * add more tests * more fixes. * figure out a way to pass lora_scale. * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * unify the retrieval logic of lora_scale. * move adjust_lora_scale_text_encoder to lora.py. * introduce dynamic adjustment lora scale support to sd * fix up copies * Empty-Commit * add: test to check fusion equivalence on different scales. * handle lora fusion warning. * make lora smaller * make lora smaller * make lora smaller --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

sayakpaul added 4 commits August 30, 2023 10:14

throw warning when more than one lora is attempted to be fused.

91b34ab

introduce support of lora scale during fusion.

c9eeb78

change test name

37692b1

changes

cfd19a5

sayakpaul requested a review from patrickvonplaten August 30, 2023 07:17

sayakpaul added 13 commits August 30, 2023 15:44

change to _lora_scale

8a9dad0

lora_scale to call whenever applicable.

ed3b37a

debugging

b86a8f6

lora_scale additional.

80839d6

cross_attention_kwargs

2ed1f2a

lora_scale -> scale.

3967da8

lora_scale fix

e24fd70

lora_scale in patched projection.

21e765b

debugging

9678ed2

debugging

acbbb4d

debugging

0c2dad4

debugging

6269412

debugging

cc0c7ec

sayakpaul added 8 commits August 30, 2023 18:48

debugging

b357ffc

debugging

8495b43

debugging

96fc1af

debugging

0c501d3

debugging

016d3e9

debugging

910e96b

debugging

de159da

debugging

4ee8dbf

patrickvonplaten reviewed Sep 1, 2023

View reviewed changes

src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Sep 1, 2023

View reviewed changes

sayakpaul commented Sep 4, 2023

View reviewed changes

src/diffusers/pipelines/versatile_diffusion/modeling_text_unet.py Outdated Show resolved Hide resolved

sayakpaul and others added 7 commits September 4, 2023 09:13

Apply suggestions from code review

9705cc2

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

unify the retrieval logic of lora_scale.

bebab12

move adjust_lora_scale_text_encoder to lora.py.

81f7ddf

introduce dynamic adjustment lora scale support to sd

e2c835c

Merge branch 'main' into lora-improvements-pt3

ca48db6

fix up copies

f2026ac

Empty-Commit

7444896

patrickvonplaten approved these changes Sep 4, 2023

View reviewed changes

add: test to check fusion equivalence on different scales.

e60f450

sayakpaul mentioned this pull request Sep 4, 2023

allow passing components to connected pipelines when use the combined pipeline #4883

Merged

handle lora fusion warning.

bf1052b

patrickvonplaten added 4 commits September 4, 2023 21:08

make lora smaller

4733384

make lora smaller

dabdd58

make lora smaller

51824c7

Merge branch 'main' into lora-improvements-pt3

972c8e8

patrickvonplaten merged commit c81a88b into main Sep 4, 2023

patrickvonplaten reviewed Sep 4, 2023

View reviewed changes

patrickvonplaten deleted the lora-improvements-pt3 branch September 4, 2023 21:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Core] LoRA improvements pt. 3 #4842

[Core] LoRA improvements pt. 3 #4842

Uh oh!

sayakpaul commented Aug 30, 2023

Uh oh!

HuggingFaceDocBuilderDev commented Aug 30, 2023 •

edited

Loading

Uh oh!

apolinario commented Aug 30, 2023 •

edited

Loading

Uh oh!

sayakpaul commented Aug 30, 2023

Uh oh!

sayakpaul commented Aug 30, 2023

Uh oh!

Uh oh!

patrickvonplaten left a comment

Uh oh!

Uh oh!

patrickvonplaten left a comment

Uh oh!

sayakpaul commented Sep 4, 2023

Uh oh!

sayakpaul commented Sep 4, 2023 •

edited

Loading

Uh oh!

sayakpaul commented Sep 4, 2023

Uh oh!

sayakpaul commented Sep 4, 2023 •

edited

Loading

Uh oh!

patrickvonplaten Sep 4, 2023

Uh oh!

patrickvonplaten commented Sep 4, 2023

Uh oh!

xhinker commented Sep 23, 2023

Uh oh!

sayakpaul commented Sep 23, 2023

Uh oh!

Uh oh!

[Core] LoRA improvements pt. 3 #4842

[Core] LoRA improvements pt. 3 #4842

Uh oh!

Conversation

sayakpaul commented Aug 30, 2023

What does this PR do?

Uh oh!

HuggingFaceDocBuilderDev commented Aug 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apolinario commented Aug 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Aug 30, 2023

Uh oh!

sayakpaul commented Aug 30, 2023

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented Sep 4, 2023

Uh oh!

sayakpaul commented Sep 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Sep 4, 2023

Uh oh!

sayakpaul commented Sep 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patrickvonplaten Sep 4, 2023

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Sep 4, 2023

Uh oh!

xhinker commented Sep 23, 2023

Uh oh!

sayakpaul commented Sep 23, 2023

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Aug 30, 2023 •

edited

Loading

apolinario commented Aug 30, 2023 •

edited

Loading

sayakpaul commented Sep 4, 2023 •

edited

Loading

sayakpaul commented Sep 4, 2023 •

edited

Loading