Repository navigation
from_single_file() support for AnimaTextConditioner #14930
Description
Activity
For some weird reason, if I try adding single file support for AnimaTextConditioner myself, I do not get the same output. This happens even if I point it to this exact file here.
I already compared the state_dict, and it appears to be identical, so I guess it has something to do with the initialization happening differently somehow, but I have no idea how to debug this.
The following code snippet should be enough to reproduce the problem, with the safetensors file in the attached zip folder:
from diffusers import AnimaTextConditioner from safetensors.torch import load_file import torch input_dict_pretrained = load_file(r"llm_adapter from_pretrained input.safetensors") device = "cpu" inputs = {key: tensor.to(device) for key, tensor in input_dict_pretrained.items()} text_conditioner1 = AnimaTextConditioner.from_pretrained( "circlestone-labs/Anima-Base-v1.0-Diffusers", subfolder="text_conditioner", dtype=torch.bfloat16, # low_cpu_mem_usage=True ) text_conditioner2 = AnimaTextConditioner.from_single_file( r"https://huggingface.co/circlestone-labs/Anima-Base-v1.0-Diffusers/blob/main/text_conditioner/diffusion_pytorch_model.safetensors", config="circlestone-labs/Anima-Base-v1.0-Diffusers", subfolder="text_conditioner", dtype=torch.bfloat16, # low_cpu_mem_usage=True ) # text_conditioner1.to(torch.bfloat16) # text_conditioner2.to(torch.bfloat16) text_conditioner1.to(device) text_conditioner2.to(device) with torch.inference_mode(): output1 = text_conditioner1(**inputs) output2 = text_conditioner2(**inputs) print(torch.equal(output1, output2))
llm_adapter from_pretrained input.zip
If I just run the code as-is, I get different outputs, but if I uncomment the line
test_conditioner1.to(torch.bfloat16), I get identical outputs. And I get the warning about casting to bfloat16 using.to()if I specifydtype=torch.bfloat16in the.from_single_file()method, but not the.from_pretrained()method.When I run the code through a debugger, I notice that
.from_pretrained()initializes the model in bfloat16, and does not cast to bfloat16 via.to(torch.bfloat16), while.from_single_file()does the type casting directly.
Is your feature request related to a problem? Please describe.
Although the Anima developer has recommended that community finetuners avoid training the
llm_adapterpart of the model, it appears that at least some of them have been doing so, which means that loading only thetransformerportion of their finetuned models might not produce expected results.In fact, I compared the
llm_adaptervalues of the official base-v1.0 model and the turbo-v1.1 model, and they are in fact not identical. The difference seems to be down to converting to FP16 and back to BF16, but still I think it is best to add this functionality.Describe the solution you'd like.
I would like to be able to do this:
Describe alternatives you've considered.
I guess it is possible to use this script to manually convert the
llm_adapterportion, then callAnimaTextConditioner.from_pretrained(), but this will be very clunky.Additional context.
Add any other context or screenshots about the feature request here.