【Hackathon 5th No.84】新增模型视频生成模型MS-Image2Video+MS-Vid2Vid-XL #345

Yang-Changhui · 2023-12-08T07:21:13Z

目前已完成图生视频，已加入pipeline,请问权重和配置文件上传到哪里

paddle-bot · 2023-12-08T07:21:18Z

Thanks for your contribution!

westfish · 2023-12-18T06:24:43Z

目前已完成图生视频，已加入pipeline,请问权重和配置文件上传到哪里

你好，可以提供一个权重的网盘链接，我们可以帮你上传到bos上

westfish · 2023-12-18T06:27:38Z

models下的这些新增的模块，还请额外增加前缀，如modelscope_，以此区分现有的一些模块

Yang-Changhui · 2023-12-18T12:24:04Z

目前已完成图生视频，已加入pipeline,请问权重和配置文件上传到哪里

你好，可以提供一个权重的网盘链接，我们可以帮你上传到bos上
你好，百度网盘无法上传大于4G的文件，压缩后权重文件还是大于4G；阿里云盘可以上传，但是不能分享大于2G的文件，请问是否可以分享ai studio项目链接

westfish · 2023-12-19T06:03:45Z

目前已完成图生视频，已加入pipeline,请问权重和配置文件上传到哪里

你好，可以提供一个权重的网盘链接，我们可以帮你上传到bos上
你好，百度网盘无法上传大于4G的文件，压缩后权重文件还是大于4G；阿里云盘可以上传，但是不能分享大于2G的文件，请问是否可以分享ai studio项目链接

可以的

Yang-Changhui · 2023-12-19T08:04:47Z

目前已完成图生视频，已加入pipeline,请问权重和配置文件上传到哪里

你好，可以提供一个权重的网盘链接，我们可以帮你上传到bos上
你好，百度网盘无法上传大于4G的文件，压缩后权重文件还是大于4G；阿里云盘可以上传，但是不能分享大于2G的文件，请问是否可以分享ai studio项目链接

可以的

ai studio生成版本不能超过1G，这又该怎么办啊，

westfish · 2023-12-20T16:40:07Z

目前已完成图生视频，已加入pipeline,请问权重和配置文件上传到哪里

你好，可以提供一个权重的网盘链接，我们可以帮你上传到bos上

你好，百度网盘无法上传大于4G的文件，压缩后权重文件还是大于4G；阿里云盘可以上传，但是不能分享大于2G的文件，请问是否可以分享ai studio项目链接

可以的

ai studio生成版本不能超过1G，这又该怎么办啊，

你可以上传到huggingface上，然后把相关链接分享给我，也是可以的

shiyutang · 2023-12-21T02:41:57Z

@westfish 请问CI需要看下么～

LokeZhou · 2023-12-21T02:47:15Z

@westfish 请问CI需要看下么～

已rerun

westfish · 2023-12-22T03:08:18Z

请问vid2vid任务完成的怎么样了呀，快到时间点了

Yang-Changhui · 2023-12-22T03:10:13Z

请问vid2vid任务完成的怎么样了呀，快到时间点了

这个还没有完成

Yang-Changhui · 2023-12-28T02:34:22Z

请问vid2vid任务完成的怎么样了呀，快到时间点了

你好，vid2vid任务已完成，与原始模型有误差，具体问题已提交issue,PaddlePaddle/Paddle#60293

westfish · 2023-12-28T11:10:59Z

权重文件准备好了吗

Yang-Changhui · 2023-12-28T11:58:13Z

权重文件准备好了吗
权重文件太大了，我上传不了huggingface，你可不可以登陆我的ai studio账号，进去下载啊

westfish · 2023-12-29T03:10:21Z

公开到这里试试看呢 https://aistudio.baidu.com/modelsoverview

Yang-Changhui · 2023-12-29T03:22:18Z

公开到这里试试看呢 https://aistudio.baidu.com/modelsoverview

这里只能上传1M以内的文件，因为我上传的模型是通过利用ai studio的数据集上传后使用的，我把数据集公开后，你从哪里下是不是也可以，
模型文件下载地址：https://aistudio.baidu.com/datasetdetail/246052

westfish · 2023-12-29T07:13:59Z

请提供完整的推理脚本，可以上传到这个目录：ppdiffusers/tests/community

Yang-Changhui · 2023-12-29T07:17:39Z

请提供完整的推理脚本，可以上传到这个目录：ppdiffusers/tests/community

在上传的两个pipeline中，都有example，这个可以吗

westfish · 2023-12-29T08:32:22Z

提供的文件貌似有问题

westfish · 2023-12-29T08:33:11Z

Yang-Changhui · 2023-12-29T08:35:47Z

提供的文件貌似有问题

请问，你运行的是video-to-video任务吗，如果是的话是正常的，因为这个模型中即包含了vision-transformer权重，也包含了text-transformer权重，只是在这个任务中并没有用到vision模块

westfish · 2023-12-29T08:41:56Z

ppdiffusers/ppdiffusers/pipelines/img_to_video/pipeline_img_to_video.py

+        noise = self.build_noise()
+
+        model_kwargs = [{"y": img_embedding, "fps": self.fps_tensor}, {"y": self.zero_feature, "fps": self.fps_tensor}]
+        state_dict = paddle.load("/home/aistudio/img_to_video/unet/model_state.pdparams")


需要删除

westfish · 2024-01-03T02:57:46Z

video2video运行到时候会遇到这个问题：

del from a BertForSequenceClassification model).
Traceback (most recent call last):
File "/root/paddlejob/workspace/zhangxu/project/upgrade/hackathon/pipe_video_to_video.py", line 7, in
video_frames = pipe(prompt=prompt, video_path=video_path).frames
File "/root/paddlejob/workspace/zhangxu/miniconda3/envs/paddle-develop/lib/python3.10/site-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "/root/paddlejob/workspace/zhangxu/miniconda3/envs/paddle-develop/lib/python3.10/site-packages/paddle/base/dygraph/base.py", line 352, in _decorate_function
return func(*args, **kwargs)
File "/root/paddlejob/workspace/zhangxu/project/upgrade/hackathon/PaddleMIX/ppdiffusers/ppdiffusers/pipelines/video_to_video/pipeline_video_to_video.py", line 329, in call
input = self.input_preprocess(
File "/root/paddlejob/workspace/zhangxu/project/upgrade/hackathon/PaddleMIX/ppdiffusers/ppdiffusers/pipelines/video_to_video/pipeline_video_to_video.py", line 267, in input_preprocess
stride = round(_fps / sample_fps)
ZeroDivisionError: float division by zero

Yang-Changhui · 2024-01-03T03:01:49Z

video2video运行到时候会遇到这个问题：

del from a BertForSequenceClassification model). Traceback (most recent call last): File "/root/paddlejob/workspace/zhangxu/project/upgrade/hackathon/pipe_video_to_video.py", line 7, in video_frames = pipe(prompt=prompt, video_path=video_path).frames File "/root/paddlejob/workspace/zhangxu/miniconda3/envs/paddle-develop/lib/python3.10/site-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), **kw) File "/root/paddlejob/workspace/zhangxu/miniconda3/envs/paddle-develop/lib/python3.10/site-packages/paddle/base/dygraph/base.py", line 352, in _decorate_function return func(*args, **kwargs) File "/root/paddlejob/workspace/zhangxu/project/upgrade/hackathon/PaddleMIX/ppdiffusers/ppdiffusers/pipelines/video_to_video/pipeline_video_to_video.py", line 329, in call input = self.input_preprocess( File "/root/paddlejob/workspace/zhangxu/project/upgrade/hackathon/PaddleMIX/ppdiffusers/ppdiffusers/pipelines/video_to_video/pipeline_video_to_video.py", line 267, in input_preprocess stride = round(_fps / sample_fps) ZeroDivisionError: float division by zero

你好，你这个问题好像是没有加载视频吧

westfish · 2024-01-03T04:13:53Z

ppdiffusers/ppdiffusers/__init__.py

@@ -57,14 +57,19 @@
 else:
    from .models import (
        AsymmetricAutoencoderKL,
+        Vid2VidSDUNet,


请按照字母顺序排列，不然会太乱影响后续开发

westfish · 2024-01-03T08:14:05Z

模型已经上传，对应id为"Yang-Changhui/img-to-video-paddle"和"Yang-Changhui/video-to-video-paddle"
使用方法如下：

import paddle
from PIL import Image
from ppdiffusers import ImgToVideoSDPipeline
from ppdiffusers.utils import export_to_video, load_image

pipe = ImgToVideoSDPipeline.from_pretrained(
    "Yang-Changhui/img-to-video-paddle", paddle_dtype=paddle.float32
)
img = load_image(
    "https://paddlenlp.bj.bcebos.com/models/community/westfish/develop-upgrade0193/stable_diffusion_inpaint_boy.png"
)
# img = Image.open('test.jpg')
video_frames = pipe(img).frames
video_path = export_to_video(video_frames, output_video_path="test.mp4")

import paddle
from ppdiffusers import VideoToVideoModelscopePipeline
from ppdiffusers.utils import export_to_video
pipe = VideoToVideoModelscopePipeline.from_pretrained("Yang-Changhui/video-to-video-paddle")
video_path = 'test.mp4'
prompt = "A man is walking on the street."
video_frames = pipe(prompt=prompt, video_path=video_path).frames
video_path = export_to_video(video_frames, "video2video_test.mp4")

Yang-Changhui · 2024-01-03T08:19:31Z

模型已经上传，对应id为"Yang-Changhui/img-to-video-paddle"和"Yang-Changhui/video-to-video-paddle" 使用方法如下：

import paddle
from PIL import Image
from ppdiffusers import ImgToVideoSDPipeline
from ppdiffusers.utils import export_to_video, load_image

pipe = ImgToVideoSDPipeline.from_pretrained(
    "Yang-Changhui/img-to-video-paddle", paddle_dtype=paddle.float32
)
img = load_image(
    "https://paddlenlp.bj.bcebos.com/models/community/westfish/develop-upgrade0193/stable_diffusion_inpaint_boy.png"
)
# img = Image.open('test.jpg')
video_frames = pipe(img).frames
video_path = export_to_video(video_frames, output_video_path="test.mp4")

import paddle
from ppdiffusers import VideoToVideoModelscopePipeline
from ppdiffusers.utils import export_to_video
pipe = VideoToVideoModelscopePipeline.from_pretrained("Yang-Changhui/video-to-video-paddle")
video_path = 'test.mp4'
prompt = "A man is walking on the street."
video_frames = pipe(prompt=prompt, video_path=video_path).frames
video_path = export_to_video(video_frames, "video2video_test.mp4")

那请问接下来还需要做什么，写gradio demo吗

westfish · 2024-01-03T08:38:10Z

模型已经上传，对应id为"Yang-Changhui/img-to-video-paddle"和"Yang-Changhui/video-to-video-paddle" 使用方法如下：

import paddle
from PIL import Image
from ppdiffusers import ImgToVideoSDPipeline
from ppdiffusers.utils import export_to_video, load_image

pipe = ImgToVideoSDPipeline.from_pretrained(
    "Yang-Changhui/img-to-video-paddle", paddle_dtype=paddle.float32
)
img = load_image(
    "https://paddlenlp.bj.bcebos.com/models/community/westfish/develop-upgrade0193/stable_diffusion_inpaint_boy.png"
)
# img = Image.open('test.jpg')
video_frames = pipe(img).frames
video_path = export_to_video(video_frames, output_video_path="test.mp4")

import paddle
from ppdiffusers import VideoToVideoModelscopePipeline
from ppdiffusers.utils import export_to_video
pipe = VideoToVideoModelscopePipeline.from_pretrained("Yang-Changhui/video-to-video-paddle")
video_path = 'test.mp4'
prompt = "A man is walking on the street."
video_frames = pipe(prompt=prompt, video_path=video_path).frames
video_path = export_to_video(video_frames, "video2video_test.mp4")

那请问接下来还需要做什么，写gradio demo吗

是的，还请辛苦在ppdiffusers/examples/inference/I2VGen-XL-gradio下加入相关gradio，类似这种https://modelscope.cn/studios/damo/I2VGen-XL-Demo/summary

Yang-Changhui · 2024-01-04T11:35:22Z

@westfish 你好，gradio已上传，但是我发现使用gradio由图像生成的视频和使用代码生成的视频不一样，这种情况是什么问题呢

westfish · 2024-01-04T11:54:46Z

要确定随机种子，参数，还有输入的图片（是不是大小变了？）等都一致；你这个pipeline里没有把generaotr暴露出来，而是用的vae的seed，不知道哪里会不会又问题，你先多跑几次看看代码生成的视频是不是都一样先

Yang-Changhui · 2024-01-05T04:46:58Z

@westfish 你好，unet模型的名字起错了，应该是STUnet，我的命名时SFUnet,已经修改了，能麻烦你将两个配置文件中的unet和model_index.json文件重新上传一下吗，https://aistudio.baidu.com/datasetdetail/246052 ，或者还是用原来的呢

westfish · 2024-01-05T06:32:23Z

@westfish 你好，unet模型的名字起错了，应该是STUnet，我的命名时SFUnet,已经修改了，能麻烦你将两个配置文件中的unet和model_index.json文件重新上传一下吗，https://aistudio.baidu.com/datasetdetail/246052 ，或者还是用原来的呢

嗯嗯帮你重新上传bos了

Yang-Changhui · 2024-01-05T08:27:20Z

要确定随机种子，参数，还有输入的图片（是不是大小变了？）等都一致；你这个pipeline里没有把generaotr暴露出来，而是用的vae的seed，不知道哪里会不会又问题，你先多跑几次看看代码生成的视频是不是都一样先

这个是没有问题的，代码由图生成的视频都是一样的，视频生成视频V100 32G会显示超内存，之前就不会有这个问题，使用的是相同的代码，

westfish · 2024-01-08T03:46:03Z

嗯嗯目前相关工作都搞好了吧现在给你合入吧

westfish

LGTM

…ddle#345) 目前已完成图生视频，已加入pipeline,请问权重和配置文件上传到哪里

img-to-video

1cb2ff0

paddle-bot bot added the contributor label Dec 8, 2023

shiyutang requested a review from JunnYu December 8, 2023 09:58

luotao1 mentioned this pull request Dec 8, 2023

【PaddlePaddle Hackathon 5th】开源贡献个人挑战赛 PaddlePaddle/Paddle#57262

Open

add-img-to-video

f3e3761

westfish self-requested a review December 21, 2023 02:47

Yang-Changhui added 2 commits December 28, 2023 10:20

add_video2video

02f26ef

add-video2video

eb1844b

westfish reviewed Jan 3, 2024

View reviewed changes

Update pipeline_img_to_video.py

14caf04

westfish reviewed Jan 3, 2024

View reviewed changes

Yang-Changhui added 5 commits January 3, 2024 13:32

Update __init__.py

7931531

Update __init__.py

fa56869

Update __init__.py

70c3169

Update __init__.py

ba2b4e0

Update __init__.py

2d48c8c

Yang-Changhui added 2 commits January 4, 2024 19:19

add-img-to-video

896b555

add-img-to-video

9a943fb

add-img-to-video

561918f

westfish approved these changes Jan 8, 2024

View reviewed changes

westfish merged commit ff0d2f2 into PaddlePaddle:develop Jan 8, 2024
2 of 3 checks passed

【Hackathon 5th No.84】新增模型视频生成模型MS-Image2Video+MS-Vid2Vid-XL #345

【Hackathon 5th No.84】新增模型视频生成模型MS-Image2Video+MS-Vid2Vid-XL #345

Conversation

Yang-Changhui commented Dec 8, 2023 • edited Loading

paddle-bot bot commented Dec 8, 2023

westfish commented Dec 18, 2023

westfish commented Dec 18, 2023

Yang-Changhui commented Dec 18, 2023

westfish commented Dec 19, 2023

Yang-Changhui commented Dec 19, 2023

westfish commented Dec 20, 2023

shiyutang commented Dec 21, 2023

LokeZhou commented Dec 21, 2023

westfish commented Dec 22, 2023

Yang-Changhui commented Dec 22, 2023

Yang-Changhui commented Dec 28, 2023 • edited Loading

westfish commented Dec 28, 2023

Yang-Changhui commented Dec 28, 2023

westfish commented Dec 29, 2023

Yang-Changhui commented Dec 29, 2023 • edited Loading

westfish commented Dec 29, 2023

Yang-Changhui commented Dec 29, 2023

westfish commented Dec 29, 2023

westfish commented Dec 29, 2023

Yang-Changhui commented Dec 29, 2023

westfish Dec 29, 2023

Choose a reason for hiding this comment

westfish commented Jan 3, 2024

Yang-Changhui commented Jan 3, 2024 • edited Loading

westfish Jan 3, 2024 • edited Loading

Choose a reason for hiding this comment

westfish commented Jan 3, 2024

Yang-Changhui commented Jan 3, 2024

westfish commented Jan 3, 2024

Yang-Changhui commented Jan 4, 2024

westfish commented Jan 4, 2024

Yang-Changhui commented Jan 5, 2024 • edited Loading

westfish commented Jan 5, 2024

Yang-Changhui commented Jan 5, 2024

westfish commented Jan 8, 2024 • edited Loading

westfish left a comment

Choose a reason for hiding this comment

Yang-Changhui commented Dec 8, 2023 •

edited

Loading

Yang-Changhui commented Dec 28, 2023 •

edited

Loading

Yang-Changhui commented Dec 29, 2023 •

edited

Loading

Yang-Changhui commented Jan 3, 2024 •

edited

Loading

westfish Jan 3, 2024 •

edited

Loading

Yang-Changhui commented Jan 5, 2024 •

edited

Loading

westfish commented Jan 8, 2024 •

edited

Loading