HunyuanVideoPipeline not support enable_sequential_cpu_offload()？

[zhaoyun0071]

Describe the bug

when use pipe.enable_sequential_cpu_offload() for purpose of low vram.

but get error below and got the black video

diffusers\image_processor.py:147: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

the similar issue：#10136 (comment)

Reproduction

import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils import export_to_video

model_id = "tencent/HunyuanVideo"
transformer = HunyuanVideoTransformer3DModel.from_pretrained(
    model_id, subfolder="transformer", torch_dtype=torch.bfloat16
)
pipe = HunyuanVideoPipeline.from_pretrained(model_id, transformer=transformer, torch_dtype=torch.float16)
pipe.vae.enable_tiling()
pipe.enable_sequential_cpu_offload()

output = pipe(
    prompt="A cat walks on the grass, realistic",
    height=320,
    width=512,
    num_frames=61,
    num_inference_steps=30,
).frames[0]
export_to_video(output, "output.mp4", fps=15)

Logs

No response

System Info

windows 11
RTX3090
diffusers 0.32.0.dev 5fb3a98
torch 2.2.2 cu118

Who can help?

@DN6 @a-r-r-o-w

[hlky]

Could you try with latest PyTorch version? Also what's the driver version from nvidia-smi?

[DN6]

@zhaoyun0071 Could you also try setting the dtype to torch.bfloat16 in the Pipeline? It is currently set to torch.float16 while the transformer is in torch.bfloat16. It could be causing NaNs when decoding latents.

[zhaoyun0071]

@zhaoyun0071 Could you also try setting the dtype to torch.bfloat16 in the Pipeline? It is currently set to torch.float16 while the transformer is in torch.bfloat16. It could be causing NaNs when decoding latents.

https://github.com/huggingface/diffusers/blob/7667cfcb41dfeb8f217e4314dcf2d561b8ca41d2/docs/source/en/api/pipelines/hunyuan_video.md

Recommendations for inference:

Both text encoders should be in torch.float16.
Transformer should be in torch.bfloat16.
VAE should be in torch.float16.
num_frames should be of the form 4 * k + 1, for example 49 or 129.

[SHYuanBest]

@zhaoyun0071 Could you also try setting the dtype to torch.bfloat16 in the Pipeline? It is currently set to torch.float16 while the transformer is in torch.bfloat16. It could be causing NaNs when decoding latents.

"replication_pad3d_cuda" not implemented for 'BFloat16'

[SHYuanBest]

i upgrade the torch, and fix the warming

[zhaoyun0071]

i upgrade the torch, and fix the warming

solved black video problem？

[SHYuanBest]

i upgrade the torch, and fix the warming

solved black video problem？

yes, with torch==2.5.1 and without change the default dtype

output.mp4

[zhaoyun0071]

i upgrade the torch, and fix the warming

solved black video problem？

yes, with torch==2.5.1 and without change the default dtype

output.mp4

can you try my code?

import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils import export_to_video

model_id = "tencent/HunyuanVideo"
transformer = HunyuanVideoTransformer3DModel.from_pretrained(
    model_id, subfolder="transformer", torch_dtype=torch.bfloat16
)
pipe = HunyuanVideoPipeline.from_pretrained(model_id, transformer=transformer, torch_dtype=torch.float16)
pipe.vae.enable_tiling()
pipe.enable_sequential_cpu_offload()

output = pipe(
    prompt="A cat walks on the grass, realistic",
    height=320,
    width=512,
    num_frames=61,
    num_inference_steps=30,
).frames[0]
export_to_video(output, "output.mp4", fps=15)

[SHYuanBest]

i upgrade the torch, and fix the warming

solved black video problem？

yes, with torch==2.5.1 and without change the default dtype
output.mp4

can you try my code?

import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils import export_to_video

model_id = "tencent/HunyuanVideo"
transformer = HunyuanVideoTransformer3DModel.from_pretrained(
    model_id, subfolder="transformer", torch_dtype=torch.bfloat16
)
pipe = HunyuanVideoPipeline.from_pretrained(model_id, transformer=transformer, torch_dtype=torch.float16)
pipe.vae.enable_tiling()
pipe.enable_sequential_cpu_offload()

output = pipe(
    prompt="A cat walks on the grass, realistic",
    height=320,
    width=512,
    num_frames=61,
    num_inference_steps=30,
).frames[0]
export_to_video(output, "output.mp4", fps=15)

output.mp4

[zhaoyun0071]

i upgrade the torch, and fix the warming

solved black video problem？

yes, with torch==2.5.1 and without change the default dtype
output.mp4

can you try my code?

import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils import export_to_video

model_id = "tencent/HunyuanVideo"
transformer = HunyuanVideoTransformer3DModel.from_pretrained(
    model_id, subfolder="transformer", torch_dtype=torch.bfloat16
)
pipe = HunyuanVideoPipeline.from_pretrained(model_id, transformer=transformer, torch_dtype=torch.float16)
pipe.vae.enable_tiling()
pipe.enable_sequential_cpu_offload()

output = pipe(
    prompt="A cat walks on the grass, realistic",
    height=320,
    width=512,
    num_frames=61,
    num_inference_steps=30,
).frames[0]
export_to_video(output, "output.mp4", fps=15)

output.mp4

many thanks

[a-r-r-o-w]

Awesome, glad to see it was resolved. Marking this as closed. Just to summarize, it was a problem with torch version and not something to do with enable_sequential_cpu_offload

[SahilCarterr]

Got this error while this running code on colab
@SHYuanBest

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_http.py](https://localhost:8080/#) in hf_raise_for_status(response, endpoint_name)
    405     try:
--> 406         response.raise_for_status()
    407     except HTTPError as e:

15 frames
HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/tencent/HunyuanVideo/resolve/main/transformer/config.json

The above exception was the direct cause of the following exception:

EntryNotFoundError                        Traceback (most recent call last)
EntryNotFoundError: 404 Client Error. (Request ID: Root=1-6762d4fe-1a2ff7e2457299f12d6c85e1;5cbc5535-e9f1-45b4-b52b-0af9f58a91e3)

Entry Not Found for url: https://huggingface.co/tencent/HunyuanVideo/resolve/main/transformer/config.json.

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/diffusers/configuration_utils.py](https://localhost:8080/#) in load_config(cls, pretrained_model_name_or_path, return_unused_kwargs, return_commit_hash, **kwargs)
    404                 )
    405             except EntryNotFoundError:
--> 406                 raise EnvironmentError(
    407                     f"{pretrained_model_name_or_path} does not appear to have a file named {cls.config_name}."
    408                 )

OSError: tencent/HunyuanVideo does not appear to have a file named config.json.

[hlky]

@SahilCarterr add revision='refs/pr/18' to from_pretrained until weights are merged.