Lora OneTrainer for Hunyuan video compatible with ComfyUI?

[ivaxsirc]

Your question

Is it possible that Loras created with OneTrainer for Hunyuan video are not compatible with ComfyUI or cannot be interpreted by it, yet?

Thanks

Logs

Other

No response

[arcteryox]

Here is the list of keys not loaded by ComfyUI when trying to use a Hunyuan Lora/Dora trained with OneTrainer:

ot_hv_lora_dora_keys.txt

[Calamdor]

FYI from the OT Discord:
Found a workaround. Find the lora.py file in ComfyUI and replace the code in if isinstance(model, comfy.model_base.HunyuanVideo):
with the code in if isinstance(model, comfy.model_base.Flux)

As Nergoar used the Flux Lora format for the Hunyuan Lora, this sounds correct.

[AcademiaSD]

try this:

if isinstance(model, comfy.model_base.HunyuanVideo):
    diffusers_keys = comfy.utils.flux_to_diffusers(model.model_config.unet_config, output_prefix="diffusion_model.")
    for j in diffusers_keys:
        if j.endswith(".weight"):
            to = diffusers_keys[j]
            key_map["transformer.{}".format(j[:-len(".weight")])] = to #simpletrainer and probably regular diffusers flux lora format
            key_map["lycoris_{}".format(j[:-len(".weight")].replace(".", "_"))] = to #simpletrainer lycoris
            key_map["lora_transformer_{}".format(j[:-len(".weight")].replace(".", "_"))] = to #onetrainer
    
    for k in sdk:
        if k.startswith("diffusion_model.") and k.endswith(".weight"):
            # diffusion-pipe lora format
            key_lora = k
            key_lora = key_lora.replace("_mod.lin.", "_mod.linear.").replace("_attn.qkv.", "_attn_qkv.").replace("_attn.proj.", "_attn_proj.")
            key_lora = key_lora.replace("mlp.0.", "mlp.fc1.").replace("mlp.2.", "mlp.fc2.")
            key_lora = key_lora.replace(".modulation.lin.", ".modulation.linear.")
            key_lora = key_lora[len("diffusion_model."):-len(".weight")]
            key_map["transformer.{}".format(key_lora)] = k
            key_map["diffusion_model.{}".format(key_lora)] = k  # Old loras

[Calamdor]

try this:

if isinstance(model, comfy.model_base.HunyuanVideo):
    diffusers_keys = comfy.utils.flux_to_diffusers(model.model_config.unet_config, output_prefix="diffusion_model.")
    for j in diffusers_keys:
        if j.endswith(".weight"):
            to = diffusers_keys[j]
            key_map["transformer.{}".format(j[:-len(".weight")])] = to #simpletrainer and probably regular diffusers flux lora format
            key_map["lycoris_{}".format(j[:-len(".weight")].replace(".", "_"))] = to #simpletrainer lycoris
            key_map["lora_transformer_{}".format(j[:-len(".weight")].replace(".", "_"))] = to #onetrainer
    
    for k in sdk:
        if k.startswith("diffusion_model.") and k.endswith(".weight"):
            # diffusion-pipe lora format
            key_lora = k
            key_lora = key_lora.replace("_mod.lin.", "_mod.linear.").replace("_attn.qkv.", "_attn_qkv.").replace("_attn.proj.", "_attn_proj.")
            key_lora = key_lora.replace("mlp.0.", "mlp.fc1.").replace("mlp.2.", "mlp.fc2.")
            key_lora = key_lora.replace(".modulation.lin.", ".modulation.linear.")
            key_lora = key_lora[len("diffusion_model."):-len(".weight")]
            key_map["transformer.{}".format(key_lora)] = k
            key_map["diffusion_model.{}".format(key_lora)] = k  # Old loras

Can confirm in Swarm (editing the correct Lora.py in the comfy backend):
No longer get errors about the keys in the log.
Resulting video with Lora loaded has changed resulting video (same seed, Lora loaded/unloaded) and has resemblance to the character I trained in OneTrainer.

Unsure if this is the proper solution for a PR or not, however.

[Calamdor]

Also - Dora keys from OT are not loaded:
I have tried a few hacks to make it work but no luck.

lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_k.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_k.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_k.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_k.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_out_0.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_out_0.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_out_0.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_out_0.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_q.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_q.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_q.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_q.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_v.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_v.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_v.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_v.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_0_proj.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_0_proj.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_0_proj.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_0_proj.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_2.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_2.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_2.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_2.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_k.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_k.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_k.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_k.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_out_0.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_out_0.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_out_0.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_out_0.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_q.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_q.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_q.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_q.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_v.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_v.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_v.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_v.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_0_proj.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_0_proj.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_0_proj.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_0_proj.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_2.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_2.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_2.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_2.lora_up.weight

[AcademiaSD]

Yes, those notices appear, but the video lora works well, if you look closely the number of notices is much smaller

[ivaxsirc]

Yes, those notices appear, but the video lora works well, if you look closely the number of notices is much smaller

Yes, it's true i tested and notices is smaller, and the most important the lora works

[KodyAG]

Also - Dora keys from OT are not loaded: I have tried a few hacks to make it work but no luck.

lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_k.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_k.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_k.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_k.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_out_0.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_out_0.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_out_0.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_out_0.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_q.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_q.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_q.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_q.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_v.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_v.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_v.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_attn_to_v.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_0_proj.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_0_proj.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_0_proj.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_0_proj.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_2.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_2.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_2.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_0_ff_net_2.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_k.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_k.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_k.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_k.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_out_0.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_out_0.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_out_0.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_out_0.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_q.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_q.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_q.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_q.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_v.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_v.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_v.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_attn_to_v.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_0_proj.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_0_proj.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_0_proj.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_0_proj.lora_up.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_2.alpha
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_2.dora_scale
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_2.lora_down.weight
lora key not loaded: lora_transformer_context_embedder_token_refiner_refiner_blocks_1_ff_net_2.lora_up.weight

Well I modified the lora.py file and can confirm my DORA from OT seems to be working, despite the lora keys not loaded notice, though it is shortened now