Description
Is your feature request related to a problem? Please describe.
X-Labs was the first lab to add FLUX.1 LoRA training (https://github.com/XLabs-AI/x-flux), however the keys are not working with diffusers (silent erroring - I just caught because I tried after investigating #9291), it doesn't error but the style isn't applied - the results with or without the LoRA are identical)
Describe the solution you'd like.
Support X-Labs trained keys natively
Additional context.
There is a community script that converts the XLabs format to diffusers https://gist.github.com/Leommm-byte/6b331a1e9bd53271210b26543a7065d6
Also as a reference, the most popular LoRA on the Hub is trained with this library, so it's highly relevant to support them imo: https://huggingface.co/XLabs-AI/flux-RealismLora
Minimal code to reproduce
import torch
from diffusers import DiffusionPipeline
device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = DiffusionPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
pipe = pipe.to(device)
pipe.load_lora_weights("XLabs-AI/flux-RealismLora")
generator = torch.Generator(device="cpu").manual_seed(0)
prompt = "The portrait of a brazilian person"
image = pipe(prompt, generator=generator).images[0]
image
Produces the same image as without the LoRA loaded
import torch
from diffusers import DiffusionPipeline
device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = DiffusionPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
pipe = pipe.to(device)
generator = torch.Generator(device="cpu").manual_seed(0)
prompt = "The portrait of a brazilian person"
image = pipe(prompt, generator=generator).images[0]
image