[WIP]Add Wan2.2 Animate Pipeline (Continuation of #12442 by tolgacangoz) #12526

dg845 · 2025-10-21T23:02:11Z

What does this PR do?

This PR is a continuation of #12442 by @tolgacangoz. It adds a pipeline for the Wan2.2-Animate-14B model (project page, paper, code, weights), a SOTA character animation and replacement video model.

Fixes #12441 (the original requesting issue).

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yiyixuxu
@sayakpaul
@tolgacangoz

- Introduced WanAnimateTransformer3DModel and WanAnimatePipeline. - Updated get_transformer_config to handle the new model type. - Modified convert_transformer to instantiate the correct transformer based on model type. - Adjusted main execution logic to accommodate the new Animate model type.

…l guidance

…prove error handling for undefined parameters

…work for character animation and replacement - Added Wan 2.2 Animate 14B model to the documentation. - Introduced the Wan-Animate framework, detailing its capabilities for character animation and replacement. - Included example usage for the WanAnimatePipeline with preprocessing steps and guidance on input requirements.

- Introduced `WanAnimateGGUFSingleFileTests` to validate functionality. - Added dummy input generation for testing model behavior.

- Introduced `EncoderApp`, `Encoder`, `Direction`, `Synthesis`, and `Generator` classes for enhanced motion and appearance encoding. - Added `FaceEncoder`, `FaceBlock`, and `FaceAdapter` classes to integrate facial motion processing. - Updated `WanTimeTextImageMotionEmbedding` to utilize the new `Generator` for motion embedding. - Enhanced `WanAnimateTransformer3DModel` with additional face adapter and pose patch embedding for improved model functionality.

- Introduced `pad_video` method to handle padding of video frames to a target length. - Updated video processing logic to utilize the new padding method for `pose_video`, `face_video`, and conditionally for `background_video` and `mask_video`. - Ensured compatibility with existing preprocessing steps for video inputs.

…roved video processing - Added optional parameters: `conditioning_pixel_values`, `refer_pixel_values`, `refer_t_pixel_values`, `bg_pixel_values`, and `mask_pixel_values` to the `prepare_latents` method. - Updated the logic in the denoising loop to accommodate the new parameters, enhancing the flexibility and functionality of the pipeline.

…eneration - Updated the calculation of `num_latent_frames` and adjusted the shape of latent tensors to accommodate changes in frame processing. - Enhanced the `get_i2v_mask` method for better mask generation, ensuring compatibility with new tensor shapes. - Improved handling of pixel values and device management for better performance and clarity in the video processing pipeline.

…and mask generation - Consolidated the handling of `pose_latents_no_ref` to improve clarity and efficiency in latent tensor calculations. - Updated the `get_i2v_mask` method to accept batch size and adjusted tensor shapes accordingly for better compatibility. - Enhanced the logic for mask pixel values in the replacement mode, ensuring consistent processing across different scenarios.

…nced processing - Introduced custom QR decomposition and fused leaky ReLU functions for improved tensor operations. - Implemented upsampling and downsampling functions with native support for better performance. - Added new classes: `FusedLeakyReLU`, `Blur`, `ScaledLeakyReLU`, `EqualConv2d`, `EqualLinear`, and `RMSNorm` for advanced neural network layers. - Refactored `EncoderApp`, `Generator`, and `FaceBlock` classes to integrate new functionalities and improve modularity. - Updated attention mechanism to utilize `dispatch_attention_fn` for enhanced flexibility in processing.

…annotations - Removed extra-abstractioned-functions such as `custom_qr`, `fused_leaky_relu`, and `make_kernel` to streamline the codebase. - Updated class constructors and method signatures to include type hints for better clarity and type checking. - Refactored the `FusedLeakyReLU`, `Blur`, `EqualConv2d`, and `EqualLinear` classes to enhance readability and maintainability. - Simplified the `Generator` and `Encoder` classes by removing redundant parameters and improving initialization logic.

…rmer tests passing

…ible with it

…ifferent preprocessing logic than the conditioning videos

dg845 · 2025-11-10T07:31:37Z

Here are some diffusers samples starting from preprocessed inputs from the Wan2.2 official repo.

Animation:

wan_animate_video_20_step.mp4

Replacement:

wan_animate_video_replace_20_step.mp4

yiyixuxu

I left some comments.
It was a pleasure to review! really awesome work! @dg845

yiyixuxu · 2025-11-10T23:34:38Z

src/diffusers/image_processor.py

            VAE scale factor. If `do_resize` is `True`, the image is automatically resized to multiples of this factor.
-        resample (`str`, *optional*, defaults to `lanczos`):
-            Resampling filter to use when resizing the image.
+        resample (`str`, *optional*, defaults to `"lanczos"`):


can we add a new WanVaeImageProcessor(VaeImageProcessor) and put into wan folder, under utils.py file I think?
(we start to see more and more custom preprocess methods, almost everyone does and they don't really get reused across models, I think moving forward let's just do this for all new models)

cc @DN6 here too, let me know what you think

I think the changes which make _resize_and_fill and _resize_and_crop respect self.config.resample should be added to the base VaeImageProcessor class; this could also be spun off into its own PR. I agree with moving the other (Wan Animate-specific logic) into its own class.

sounds good

yiyixuxu · 2025-11-10T23:54:27Z

src/diffusers/models/transformers/transformer_wan_animate.py

+    def __repr__(self):
+        return (
+            f"{self.__class__.__name__}({self.weight.shape[1]}, {self.weight.shape[0]},"
+            f" kernel_size={self.weight.shape[2]}, stride={self.stride}, padding={self.padding})"
+        )


Suggested change

def __repr__(self):

return (

f"{self.__class__.__name__}({self.weight.shape[1]}, {self.weight.shape[0]},"

f" kernel_size={self.weight.shape[2]}, stride={self.stride}, padding={self.padding})"

)

I kind of like having this because if you print(model) then MotionConv2d will print out info similar to torch.nn.Conv2d (same with MotionLinear below). But I'm also fine with removing it.

ok if you feel strongly about keeping it!

yiyixuxu · 2025-11-11T00:03:19Z

src/diffusers/models/transformers/transformer_wan_animate.py

+    def __repr__(self):
+        return (
+            f"{self.__class__.__name__}(in_features={self.weight.shape[1]}, out_features={self.weight.shape[0]},"
+            f" bias={self.bias is not None})"
+        )


Suggested change

def __repr__(self):

return (

f"{self.__class__.__name__}(in_features={self.weight.shape[1]}, out_features={self.weight.shape[0]},"

f" bias={self.bias is not None})"

)

yiyixuxu · 2025-11-11T00:14:38Z

src/diffusers/models/transformers/transformer_wan_animate.py

+        hidden_states = hidden_states.flatten(2).transpose(1, 2)
+
+        # 3. Condition embeddings (time, text, image)
+        # timestep shape: batch_size, or batch_size, seq_len (wan 2.2 ti2v)


I think we can move one of these conditions for animate, no?

yiyixuxu · 2025-11-11T00:28:22Z

src/diffusers/models/transformers/transformer_wan_animate.py

+
+        self.gradient_checkpointing = False
+
+    def motion_batch_encode(


can we move this to forward? all the layers (motion_encoder here) should be visible in forward

yiyixuxu · 2025-11-11T00:48:36Z

src/diffusers/models/transformers/transformer_wan_animate.py

+                hidden_states = block(hidden_states, encoder_hidden_states, timestep_proj, rotary_emb)
+
+            # Face adapter integration: apply after every 5th block (0, 5, 10, 15, ...)
+            if block_idx % self.config.inject_face_latents_blocks == 0:


ohh does it make sense to create a WanAnimateTransformerBlock that includes face_adapter?
e.g.

# In __init__ self.blocks = nn.ModuleList( [ WanAnimateTransformerBlock( ...., add_face_adapter=(i % inject_face_latents_blocks == 0), ) for i in range(num_layers) ] )

@yiyixuxu my understanding is that the main advantage of creating a WanAnimateTransformerBlock would be that if there is an associated face adapter block, it will always stay on the same device as the main transformer block (since we would put WanAnimateTransformerBlock in _no_split_modules). I think the code could look something like:

class WanAnimateTransformerBlock(nn.Module): def __init__(self, ..., add_face_adapter: bool = False): super().__init__() self.transformer_block = WanTransformerBlock(...) self.face_cross_attn = None if add_face_adapter: self.face_cross_attn = WanAnimateFaceBlockCrossAttention(...) def forward(self, hidden_states, motion_vec, ...): hidden_states = self.transformer_block(hidden_states, ...) if self.face_cross_attn is not None: face_adapter_output = self.face_cross_attn(hidden_states, motion_vec) hidden_states = face_adapter_output + hidden_states return hidden_states

I think it is clearest to reuse WanTransformerBlock here so that it is easy to tell which logic is inherited from the base Wan 2.1 model and which is Wan Animate-specific. WDYT?

ohh so we are moving toward single-file format for models, so basically let's assume people who look through wan animate transformer do not need to know anything about wan transformer

i left commeenets here too https://github.com/huggingface/diffusers/pull/12526/files#r2512642810
ok if you want to re-use the block! but let's not import from the wan tranformer file

yiyixuxu · 2025-11-11T00:52:16Z

src/diffusers/models/transformers/transformer_wan_animate.py

+
+        hidden_states_original_dtype = hidden_states.dtype
+        hidden_states = self.norm_out(hidden_states.float())
+        # Move the shift and scale tensors to the same device as hidden_states.


ohh let's try to fix it here
I think all we need to do is to pack shift and scale into same layer and add that layere into _no_split_modules attribute

yiyixuxu · 2025-11-11T00:53:25Z

src/diffusers/pipelines/wan/pipeline_wan_animate.py

+        >>> face_video = load_video("path/to/face_video.mp4")
+
+        >>> # Calculate optimal dimensions based on VAE constraints
+        >>> max_area = 480 * 832


if we make a vaeimageprocessor for wan, this can be added there too

yiyixuxu · 2025-11-11T00:55:16Z

src/diffusers/pipelines/wan/pipeline_wan_animate.py

+        latents = (latents - latents_mean) * latents_recip_std
+        return latents
+
+    def destandardize_latents(self, latents: torch.Tensor) -> torch.Tensor:


can we put this back into pipeline, it is always part of pipeline

yiyixuxu · 2025-11-11T00:56:36Z

src/diffusers/pipelines/wan/pipeline_wan_animate.py

+                f" {type(prev_segment_conditioning_frames)} and value is {prev_segment_conditioning_frames}"
+            )
+
+    def standardize_latents(self, latents: torch.Tensor) -> torch.Tensor:


can we put this back into pipeline? let's try not to have too many small methods. and the normalize/denoramlize is always part of the pipeline code

yiyixuxu · 2025-11-11T02:49:01Z

src/diffusers/models/transformers/transformer_wan_animate.py

+from ..modeling_outputs import Transformer2DModelOutput
+from ..modeling_utils import ModelMixin
+from ..normalization import FP32LayerNorm
+from .transformer_wan import (


I missed a few imports here, I'm ok if you want to re-use WanTransformerBlock - but let's copy it here instead of importing from a different model file

tolgacangoz and others added 30 commits October 6, 2025 21:46

template1

3529a0a

temp2

4f2ee5e

up

778fb54

up

d77b6ba

fix-copies

2fc6ac2

style

6182d44

Refactor WanAnimate model components

8c9fd89

Enhance WanAnimatePipeline with new parameters for mode and tempora…

d01e941

…l guidance

Update WanAnimatePipeline to require additional video inputs and im…

7af953b

…prove error handling for undefined parameters

Add unit test template for WanAnimatePipeline functionality

05a01c6

Add unit tests for WanAnimateTransformer3DModel in GGUF format

22b83ce

- Introduced `WanAnimateGGUFSingleFileTests` to validate functionality. - Added dummy input generation for testing model behavior.

style

7fb6732

Update WanAnimatePipeline

624a314

style

fc0edb5

Refactor test for WanAnimatePipeline to include new input structure

eb7eedd

from einops to torch

8968b42

Merge branch 'main' into integrations/wan2.2-animate

dce83a8

style

802896e

up

84768f6

style

b8337c6

Merge branch 'main' into integrations/wan2.2-animate

4e6651b

dg845 added 25 commits October 24, 2025 06:15

Merge branch 'main' into add-wan2.2-animate-pipeline

8216aef

Add initial Wan Animate transformer tests

50329d7

Refactor face block attn into its own Attention class and fix some bugs

275d324

Fix issues (such as device placement issues) to get remaining transfo…

ac2962d

…rmer tests passing

Update Wan Animate conversion script to reflect changes to transformer

0145135

Add _repeated_blocks to Wan Animate transformer for regional compilation

bdbd141

Refactor Wan Animate pipeline to make latent preparation code more clear

2537133

Update Wan Animate pipeline tests after transformer an pipeline changes

332d3c2

Merge branch 'main' into add-wan2.2-animate-pipeline

99e56e3

Fix some batching and device placement issues in Wan Animate pipeline

00ddbb9

Remove reference_images tests for Wan Animate

1e1e706

Get Wan Animate pipeline fp16 inference tests working

a56bee1

Skip test_callback_inputs since the Wan Animate pipline is not compat…

6fb5ca8

…ible with it

Fix mask video shapes for Wan Animate replacement

1e61ed7

Use a separate VaeImageProcessor for the reference image as it uses d…

e2846f6

…ifferent preprocessing logic than the conditioning videos

Fix some more Wan Animate pipeline shape errors

3a80241

Fix more bugs in Wan Animate pipeline

86be600

Ensure that the replacement mask only has one channel

6748d25

Support Wan Animate image preprocessing, fix bugs, clean up code

f696682

Add docs for WanAnimateTransformer3DModel

80d9f8b

make style and make quality

d9c6bc6

Merge branch 'main' into add-wan2.2-animate-pipeline

4e415d3

Fix first segment I2V mask for prev segement cond latents

cbfc0ad

Use same Open CLIP checkpoint as other Wan2.1-based models

b80be86

Merge branch 'main' into add-wan2.2-animate-pipeline

d87baa5

dg845 marked this pull request as ready for review November 10, 2025 06:26

dg845 requested a review from yiyixuxu November 10, 2025 06:27

yiyixuxu reviewed Nov 11, 2025

View reviewed changes

[WIP]Add Wan2.2 Animate Pipeline (Continuation of #12442 by tolgacangoz) #12526

Are you sure you want to change the base?

[WIP]Add Wan2.2 Animate Pipeline (Continuation of #12442 by tolgacangoz) #12526

Conversation

dg845 commented Oct 21, 2025

What does this PR do?

Who can review?

Uh oh!

dg845 commented Nov 10, 2025

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dg845 Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dg845 Nov 11, 2025 •

edited

Loading