-
Notifications
You must be signed in to change notification settings - Fork 6.6k
Add ZImageImg2ImgPipeline #12751
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Add ZImageImg2ImgPipeline #12751
Conversation
Updated the pipeline structure to include ZImageImg2ImgPipeline
alongside ZImagePipeline.
Implemented the ZImageImg2ImgPipeline class for image-to-image
transformations, including necessary methods for
encoding prompts, preparing latents, and denoising.
Enhanced the auto_pipeline to map the new ZImageImg2ImgPipeline
for image generation tasks.
Added unit tests for ZImageImg2ImgPipeline to ensure
functionality and performance.
Updated dummy objects to include ZImageImg2ImgPipeline for
testing purposes.
|
For some reason the VAE Tiling couldn't meet the 0.2 diff threshold, my test has upped that to 0.3, whether further investigation is warranted I am not sure. |
asomoza
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks a lot again! for this one we should probably wait for the lora one to be merged. I left a few comments
| ) | ||
| self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor * 2) | ||
|
|
||
| def encode_prompt( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can this function be also with Copied from
| negative_prompt_embeds = [] | ||
| return prompt_embeds, negative_prompt_embeds | ||
|
|
||
| def _encode_prompt( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as before, this one can be Copied from no?
| ) | ||
| from .wan import WanImageToVideoPipeline, WanPipeline, WanVideoToVideoPipeline | ||
| from .wuerstchen import WuerstchenCombinedPipeline, WuerstchenDecoderPipeline | ||
| from .z_image import ZImageImg2ImgPipeline |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since you're adding this pipeline here, can you also add the t2i too
|
@asomoza I just thought, I have inpainting PR lined up, do you think keeping this one img2img only and inpainting after that, separately, is the better approach, to keep the PR review easier? Or is it less work for you guys if I also merge this in this PR? |
|
I prefer to keep them separated, I'm not really sure the inpainting can be good with this model so I want to test it and maybe we can add something like differential diffusion as a switch for it to be better |



What does this PR do?
This PR adds img2img pipeline for Z-Image. The summary of changes are below
Closes issue #12752
Tested using a simple script:
Testing script
Prompt:
a woman sitting in a dark room, oil painting style, impressionist, vibrant colorsLoRA functionality depends on my other PR #12750, so they will have to be merged sequentially. I did not think there was much point in leaving it out.
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@sayakpaul @asomoza