LoRa effect is none when inferencing with FluxPipeline.from_pretrained()

[DimitriosKakouris]

Ηello, I trained a LoRa with the help of the ostris/ai-toolkit repo, I believe it is based mostly on the kohya_ss repo. The LoRa saved in safetensors format when run with the sample inference code below gave me warnings on most of the LoRa keys and even though it ran fine, the output image was a black image of low size, around 1.3kB:

import torch
from diffusers import DiffusionPipeline,FluxPipeline

pipe=FluxPipeline.from_pretrained('black-forest-labs/FLUX.1-dev', torch_dtype=torch.bfloat16, device_map='balanced')
pipe.load_lora_weights("TheLastBen/The_Hound", weight_name="sandor_clegane_single_layer.safetensors")


prompt="sandor clegane drinking in a pub"

image = pipe(
    prompt=prompt,
    num_inference_steps=30,
    width=1024,
    generator=torch.Generator("cpu").manual_seed(42),
    height=1024,
).images[0]
image.save("./fluxlora/flux-lora.png")

In order to combat the incompatibility of the LoRa keys I used the script provided by the kohya_ss repo that converts his format to diffusers format convert_flux_lora.py the warnings disappeared resulting in an image that followed the prompt but without the face of "sandor clegane". It is like the effect of pipe.load_lora_weights() is non-existent. Can you help as to why this might be happening?

[asomoza]

Hi, what version of diffusers you're using? You'll need to install from source to be able to use kohya loras, recently we merged a PR that added support for them.

[DimitriosKakouris]

Hi, what version of diffusers you're using? You'll need to install from source to be able to use kohya loras, recently we merged a PR that added support for them.

Hello, I have tried both the 0.30.2 and when I install from source with pip install git+... the 0.31.dev0 version.

[asomoza]

I did a test with your code and it worked without a problem. I just removed the device_map='balanced' because I don't need it.

[DimitriosKakouris]

Okay, that is odd, It may be an issue with my environment then, I run it in an HPC cluster, some old libraries, etc. Thank you!