why some lora not effective, get snowy/blurry images

[zxf2012]

Hi, I got a problem when using lora, I downloaded a lora with safetensors from civitai, this is the link: https://civitai.com/models/121706, I used the script convert_lora_safetensor_to_diffusers.py to load lora, then i got a snowy/blurry image, but it works with stable diffusion webui；

this is my codes:

import torch
import time
from diffusers import AutoencoderKL
from diffusers import EulerAncestralDiscreteScheduler
from transformers import CLIPTextModel
from scripts import convert_lora_safetensor_to_diffusers

pipeline = None
now = int(time.time() * 1000)
model_id = "danbrown/RevAnimated-v1-2-2"

LORA_PREFIX_UNET = 'lora_unet'
LORA_PREFIX_TEXT_ENCODER = 'lora_te'
pipeline = convert_lora_safetensor_to_diffusers.convert(model_id, r"./梦幻3D场景_v1.0.safetensors", LORA_PREFIX_UNET, LORA_PREFIX_TEXT_ENCODER, 1)

pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(pipeline.scheduler.config)

text_encoder = CLIPTextModel.from_pretrained(
    model_id,
    subfolder="text_encoder",
    num_hidden_layers=10,
    torch_dtype=torch.float32
)
pipeline.text_encoder = text_encoder

pipeline.safety_checker = lambda images, clip_input: (images, None)

pipeline = pipeline.to("cuda")

prompt="changjing_3d, ((best quality)), ((masterpiece)),changjing_3d, no_humans,night,ponds, lotus leaves, lotus flowers, river lanterns, fireflies, butterflies, moonlight,close-up,depth of field"
negative_prompt="worst quality,bad quality,(monochrome:0.8)"

images = pipeline(
     prompt,
     negative_prompt=negative_prompt,
     height=800, 
     width=600,
     num_inference_steps=30,
     guidance_scale=7,
).images

for index, image in enumerate(images):
    image_name = "%s-%s" % (now, index)
    image.save("images\%s.png" % (image_name))

the result is flowing image:

but it works with webui with flowing config:

changjing_3d, ((best quality)), ((masterpiece)),changjing_3d, no_humans,night,ponds, lotus leaves, lotus flowers, river lanterns, fireflies, butterflies, moonlight,close-up,depth of field <lora:changjing_v1:1>
Negative prompt: Negative prompt: worst quality,bad quality,(monochrome:0.8)
Steps: 30, Sampler: Euler a, CFG scale: 7, Seed: 2509898195, Size: 600x800, Model hash: 4199bcdd14, Model: revAnimated_v122, Clip skip: 2, ENSD: 31337, Lora hashes: "changjing_v1: e6249feb982d", Version: v1.3.1

the result is flowing image:

Who can help me? Why is this happening?

python version:

Python: 3.9.16

CUDA version:

computer infos:

this is my requirements.txt contents:

accelerate==0.17.0
anyio==3.6.2
auth_token==0.0.3
certifi==2022.9.24
charset-normalizer==2.1.1
click==8.1.3
colorama==0.4.6
contourpy==1.0.6
cycler==0.11.0
diffusers==0.17.0
fastapi==0.96.0
filelock==3.8.0
fonttools==4.38.0
ftfy==6.1.1
h11==0.14.0
httptools==0.5.0
huggingface-hub==0.15.1
idna==3.4
importlib-metadata==5.1.0
jax==0.4.1
kiwisolver==1.4.4
matplotlib==3.6.2
ninja==1.11.1
numpy==1.23.5
opt-einsum==3.3.0
packaging==21.3
pandas==1.5.2
Pillow==8.4.0
psutil==5.9.4
pydantic==1.10.2
pyparsing==3.0.9
python-dateutil==2.8.2
python-dotenv==0.21.0
pytz==2022.6
PyYAML==6.0
regex==2022.10.31
requests==2.28.1
scipy==1.10.0
six==1.16.0
sniffio==1.3.0
starlette==0.27.0
tokenizers==0.13.2
torch==2.0.1+cu117
torchvision==0.15.2+cu117
torchaudio==2.0.2+cu117
tqdm==4.64.1
transformers==4.26.0
typing-extensions==4.4.0
urllib3==1.26.13
uvicorn==0.20.0
watchfiles==0.18.1
wcwidth==0.2.6
websockets==10.4
zipp==3.11.0
omegaconf==2.3.0
ipywidgets==8.0.6
safetensors==0.3.1
compel==1.2.1
sentencepiece==0.1.99
patch-ng==1.17.4
clip_interrogator==0.6.0

[sayakpaul]

Recommend using load_lora_weights() for these checkpoints. We don't maintain that conversion script, so don't have good handle on it.

[zxf2012]

Recommend using load_lora_weights() for these checkpoints. We don't maintain that conversion script, so don't have good handle on it.

@sayakpaul thank you for you reply, I tried load_lora_weights(), I received clear images this time, without any noise., but the lora not effective

this is my new codes:

import torch
import time
from diffusers import AutoencoderKL, EulerAncestralDiscreteScheduler, StableDiffusionPipeline
from transformers import CLIPTextModel
from compel import Compel

pipeline = None
now = int(time.time() * 1000)
torch_dtype=torch.float16

model_id = "danbrown/RevAnimated-v1-2-2"
pipeline = StableDiffusionPipeline.from_pretrained(
    model_id,
    torch_dtype=torch_dtype,
    resume_download=True,
    safety_checker=None,
)

#set vae
vae = AutoencoderKL.from_pretrained("lint/anime_vae", torch_dtype=torch_dtype)
pipeline.vae = vae

#set sampler
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(pipeline.scheduler.config)

#load lora weights
pipeline.load_lora_weights(".", weight_name="梦幻3D场景_v1.0.safetensors")

#需要在加载lora之后执行，不然会报AttributeError: 'ModuleList' object has no attribute '10_mlp_fc1'的错误
#set clip_skip
text_encoder = CLIPTextModel.from_pretrained(
    model_id,
    subfolder="text_encoder",
    num_hidden_layers=10,
    torch_dtype=torch_dtype
)
pipeline.text_encoder = text_encoder

#remove NSFW check
pipeline.safety_checker = lambda images, clip_input: (images, None)

pipeline = pipeline.to("cuda")

#set prompts
prompt="changjing_3d, ((best quality)), ((masterpiece)),changjing_3d, no_humans,night,ponds, lotus leaves, lotus flowers, river lanterns, fireflies, butterflies, moonlight,close-up,depth of field"      
negative_prompt="worst quality,bad quality,(monochrome:0.8)"

images = pipeline(
     prompt,
     negative_prompt=negative_prompt,
     height=800, 
     width=600,
     num_inference_steps=30,
     guidance_scale=7,
     cross_attention_kwargs={"scale": 1},
).images

#save the image
for index, image in enumerate(images):
    image_name = "%s-%s" % (now, index)
    image.save("images\%s.png" % (image_name))

and this is result image，it just affected by the base model "danbrown/RevAnimated-v1-2-2", the lora not effective

[sayakpaul]

Could you provide a Colab Notebook so that I can test it out?

[jexterliangsufe]

I think the problem might be that you override the text_encoder after load_lora_weights.

[zetyquickly]

I'm not be right,
but it seems like this piece of code creates PatchedLoraProjections with a particular lora_scale, and once these instances are created they don't use the scale from a kwarg cross_attention_kwargs={"scale": <scale>}, which patches self._lora_scale in here

[zxf2012]

Could you provide a Colab Notebook so that I can test it out?

this is the colab notebook:https://colab.research.google.com/drive/1bpJMfVTBWSqPxrPyQnZ1kWbh-5geZXux?usp=sharing

note: it needs you download the lora weights file manually from https://civitai.com/models/121706/dream-3d

[zxf2012]

I think the problem might be that you override the text_encoder after load_lora_weights.

If I commented out the clip_skip codes, it still has no effect.

[sayakpaul]

I'm not be right, but it seems like this piece of code creates PatchedLoraProjections with a particular lora_scale, and once these instances are created they don't use the scale from a kwarg cross_attention_kwargs={"scale": <scale>}, which patches self._lora_scale in here

@zetyquickly that only applies to the text encoders, though.

@zxf2012 not sure how much of an influence the compel-weighted prompts and CLIP skip have on the results, but even without them, I am able to see noticeable differences in the results with and without the LoRA you mentioned. Here's my Colab: https://colab.research.google.com/gist/sayakpaul/4ce4ebc3ac23695b3d8ac93b8771b3d9/issue.ipynb

[zetyquickly]

@sayakpaul yes, correct. spent an evening yesterday to figure out that the cross_attention_kwargs={"scale": <scale>} still propagates to UNET module and the scale still affects the generation result.

And it's clear that for SDv1.5 derived models, the text encoder is not properly patched with scale at load_lora_weights call. But in SDXL text encoders are patched. Is there a raised issue regarding that?

[sayakpaul]

And it's clear that for SDv1.5 derived models, the text encoder is not properly patched with scale at load_lora_weights call. But in SDXL text encoders are patched. Is there a raised issue regarding that?

Elaborate this a bit?

[zetyquickly]

This is how it loads for SDXL.
This is how it loads for SDv1.5.

My previous statement was wrong. The loading process for SDXL is the same as for SDv1.5. To apply lora_scale to text encoders in both cases, one needs to change pipe._lora_scale prior to calling load_lora_weights.

Is the issue clear here?

[chris-aeviator]

I'm getting blurry outputs with AutoTrain Advanced using SDXL https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_Dreambooth.ipynb

Kept the standard settings and only disabled gradient checkpointing. Training 500 steps on 21 images (and yes it's supermarkets ;) ).

[lhh-pi]

May I ask if the problem has been solved? I had the same problem as you.

[zxf2012]

I'm not be right, but it seems like this piece of code creates PatchedLoraProjections with a particular lora_scale, and once these instances are created they don't use the scale from a kwarg cross_attention_kwargs={"scale": <scale>}, which patches self._lora_scale in here

@zetyquickly that only applies to the text encoders, though.

@zxf2012 not sure how much of an influence the compel-weighted prompts and CLIP skip have on the results, but even without them, I am able to see noticeable differences in the results with and without the LoRA you mentioned. Here's my Colab: https://colab.research.google.com/gist/sayakpaul/4ce4ebc3ac23695b3d8ac93b8771b3d9/issue.ipynb

that's strange, I have give up on this problem, thank you for helping me with this question

[zxf2012]

May I ask if the problem has been solved? I had the same problem as you.

I have give up on this problem