-
Notifications
You must be signed in to change notification settings - Fork 6.6k
[WIP] Add Flux2 modular #12763
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
[WIP] Add Flux2 modular #12763
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
yiyixuxu
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks! I left some feedbacks, but feel free to merge once it's working:)
| ) | ||
|
|
||
|
|
||
| class Flux2AutoTextEncoderStep(AutoPipelineBlocks): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's up to you, but I think maybe we don't need to mix the remote blocks into Auto - we can separately list them so people can mix it on their own
ALL_BLOCKS = {
"text2image": TEXT2IMAGE_BLOCKS,
"image_conditioned": IMAGE_CONDITIONED_BLOCKS,
"auto": AUTO_BLOCKS,
"remote" :
}| return "Before denoise step that prepares the inputs for the denoise step in Flux2 generation." | ||
|
|
||
|
|
||
| class Flux2AutoBeforeDenoiseStep(AutoPipelineBlocks): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ohh, I think we don't need to wrap it inside a AutoPipelineBlocks if this is the only block
| ) | ||
|
|
||
|
|
||
| class Flux2AutoDenoiseStep(AutoPipelineBlocks): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same here
| ) | ||
|
|
||
|
|
||
| class Flux2AutoDecodeStep(AutoPipelineBlocks): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same here too, we don't need to wrap it inside an autoblock
if the decode step works for all workflow we esupport,
| ("before_denoise", Flux2AutoBeforeDenoiseStep()), | ||
| ("denoise", Flux2AutoDenoiseStep()), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so this is the set we use for UI, so basically we need to have these 4 blocks
- text_encoder
- vae_image_encoder
- denoise
- decode
each corresponding to a node
I would pack text_input/image_input/before_Denoise/denoise into an "AutoDenoise" block, you can pack it however you like, but I think one way to do it is to:
first, have 2 sequential blocks
for text-to-image: text_input -> before_denoise -> denoise
for image condition: text_input -> image_input -> before_denoise -> denoise
and then pack these two into an AutoDenoise
| [ | ||
| ("text_encoder", Flux2AutoTextEncoderStep()), | ||
| ("text_input", Flux2AutoTextInputStep()), | ||
| ("image_encoder", Flux2AutoVaeEncoderStep()), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| ("image_encoder", Flux2AutoVaeEncoderStep()), | |
| ("vae_image_encoder", Flux2AutoVaeEncoderStep()), |
naming it "vae_image_encoder" because there are also image_encoder blocks using like clip/siglip image encoder, just
What does this PR do?
Fixes # (issue)
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.