Add SEGA for FLUX #3
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add `remote_decode` to `remote_utils` * test dependency * test dependency * dependency * dependency * dependency * docstrings * changes * make style * apply * revert, add new options * Apply style fixes * deprecate base64, headers not needed * address comments * add license header * init test_remote_decode * more * more test * more test * skeleton for xl, flux * more test * flux test * flux packed * no scaling * -save * hunyuanvideo test * Apply style fixes * init docs * Update src/diffusers/utils/remote_utils.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * comments * Apply style fixes * comments * hybrid_inference/vae_decode * fix * tip? * tip * api reference autodoc * install tip --------- Co-authored-by: sayakpaul <spsayakpaul@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
fix-copies went uncaught it seems.
* fix-copies went uncaught it seems. * remove more unneeded encode_prompt() tests * Revert "fix-copies went uncaught it seems." This reverts commit eefb302. * empty
…eneration model (huggingface#10626) * Update EasyAnimate V5.1 * Add docs && add tests && Fix comments problems in transformer3d and vae * delete comments and remove useless import * delete process * Update EXAMPLE_DOC_STRING * rename transformer file * make fix-copies * make style * refactor pt. 1 * update toctree.yml * add model tests * Update layer_norm for norm_added_q and norm_added_k in Attention * Fix processor problem * refactor vae * Fix problem in comments * refactor tiling; remove einops dependency * fix docs path * make fix-copies * Update src/diffusers/pipelines/easyanimate/pipeline_easyanimate_control.py * update _toctree.yml * fix test * update * update * update * make fix-copies * fix tests --------- Co-authored-by: Aryan <aryan@huggingface.co> Co-authored-by: Aryan <contact.aryanvs@gmail.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
* Fix SD2.X clip single file load projection_dim Infer projection_dim from the checkpoint before loading from pretrained, override any incorrect hub config. Hub configuration for SD2.X specifies projection_dim=512 which is incorrect for SD2.X checkpoints loaded from civitai and similar. Exception was previously thrown upon attempting to load_model_dict_into_meta for SD2.X single file checkpoints. Such LDM models usually require projection_dim=1024 * convert_open_clip_checkpoint use hidden_size for text_proj_dim * convert_open_clip_checkpoint, revert checkpoint[text_proj_key].shape[1] -> [0] values are identical --------- Co-authored-by: Teriks <Teriks@users.noreply.github.com> Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
* Update pipeline_animatediff.py * Update pipeline_animatediff_controlnet.py * Update pipeline_animatediff_sparsectrl.py * Update pipeline_animatediff_video2video.py * Update pipeline_animatediff_video2video_controlnet.py --------- Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
* Add example of Ip-Adapter-Callback. * Add image links from HF Hub.
* Update ip_adapter.py * Update ip_adapter.py * Update ip_adapter.py * Update ip_adapter.py * Update ip_adapter.py * Apply style fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hlky <hlky@hlky.ac>
* initial comit * fix empty cache * fix one more * fix style * update device functions * update * update * Update src/diffusers/utils/testing_utils.py Co-authored-by: hlky <hlky@hlky.ac> * Update src/diffusers/utils/testing_utils.py Co-authored-by: hlky <hlky@hlky.ac> * Update src/diffusers/utils/testing_utils.py Co-authored-by: hlky <hlky@hlky.ac> * Update tests/pipelines/controlnet/test_controlnet.py Co-authored-by: hlky <hlky@hlky.ac> * Update src/diffusers/utils/testing_utils.py Co-authored-by: hlky <hlky@hlky.ac> * Update src/diffusers/utils/testing_utils.py Co-authored-by: hlky <hlky@hlky.ac> * Update tests/pipelines/controlnet/test_controlnet.py Co-authored-by: hlky <hlky@hlky.ac> * with gc.collect * update * make style * check_torch_dependencies * add mps empty cache * add changes * bug fix * enable on xpu * update more cases * revert * revert back * Update test_stable_diffusion_xl.py * Update tests/pipelines/stable_diffusion/test_stable_diffusion.py Co-authored-by: hlky <hlky@hlky.ac> * Update tests/pipelines/stable_diffusion/test_stable_diffusion.py Co-authored-by: hlky <hlky@hlky.ac> * Update tests/pipelines/stable_diffusion/test_stable_diffusion_img2img.py Co-authored-by: hlky <hlky@hlky.ac> * Update tests/pipelines/stable_diffusion/test_stable_diffusion_img2img.py Co-authored-by: hlky <hlky@hlky.ac> * Update tests/pipelines/stable_diffusion/test_stable_diffusion_img2img.py Co-authored-by: hlky <hlky@hlky.ac> * Apply suggestions from code review Co-authored-by: hlky <hlky@hlky.ac> * add test marker --------- Co-authored-by: hlky <hlky@hlky.ac>
* Update evaluation.md * Update docs/source/en/conceptual/evaluation.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* feat: support non-diffusers lumina2 LoRAs. * revert ipynb changes (but I don't know why this is required☹️ ) * empty --------- Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> Co-authored-by: YiYi Xu <yixu310@gmail.com>
…e#10927) * [Quantization] support pass MappingType for TorchAoConfig * Apply style fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…tization_config.py. (huggingface#10961) Update quantization_config.py
* update * refactor image-to-video pipeline * update * fix copied from * use FP32LayerNorm
…SDXL (huggingface#10951) * feat: add Mixture-of-Diffusers ControlNet Tile upscaler Pipeline for SDXL * make style make quality
* Update pipeline_cogview4.py * Use GLM instead of T5 in doc
* fix t5 training bug * Apply style fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
use style bot GH action from hfh Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
…hs` is passed in a distributed training env (huggingface#10973) * updated train_dreambooth_lora to fix the LR schedulers for `num_train_epochs` in distributed training env * fixed formatting * remove trailing newlines * fixed style error
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* update * update * addressed PR comments * update --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>
…ce#11369) * Add stochastic sampling to FlowMatchEulerDiscreteScheduler This PR adds stochastic sampling to FlowMatchEulerDiscreteScheduler based on Lightricks/LTX-Video@b1aeddd ltx_video/schedulers/rf.py * Apply style fixes * Use config value directly * Apply style fixes * Swap order * Update src/diffusers/schedulers/scheduling_flow_match_euler_discrete.py Co-authored-by: YiYi Xu <yixu310@gmail.com> * Update src/diffusers/schedulers/scheduling_flow_match_euler_discrete.py Co-authored-by: YiYi Xu <yixu310@gmail.com> --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com>
…e#11281) * initial commit * initial commit * initial commit * initial commit * initial commit * initial commit * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: Bagheera <59658056+bghira@users.noreply.github.com> * move prompt embeds, pooled embeds outside * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: hlky <hlky@hlky.ac> * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: hlky <hlky@hlky.ac> * fix import * fix import and tokenizer 4, text encoder 4 loading * te * prompt embeds * fix naming * shapes * initial commit to add HiDreamImageLoraLoaderMixin * fix init * add tests * loader * fix model input * add code example to readme * fix default max length of text encoders * prints * nullify training cond in unpatchify for temp fix to incompatible shaping of transformer output during training * smol fix * unpatchify * unpatchify * fix validation * flip pred and loss * fix shift!!! * revert unpatchify changes (for now) * smol fix * Apply style fixes * workaround moe training * workaround moe training * remove prints * to reduce some memory, keep vae in `weight_dtype` same as we have for flux (as it's the same vae) https://github.com/huggingface/diffusers/blob/bbd0c161b55ba2234304f1e6325832dd69c60565/examples/dreambooth/train_dreambooth_lora_flux.py#L1207 * refactor to align with HiDream refactor * refactor to align with HiDream refactor * refactor to align with HiDream refactor * add support for cpu offloading of text encoders * Apply style fixes * adjust lr and rank for train example * fix copies * Apply style fixes * update README * update README * update README * fix license * keep prompt2,3,4 as None in validation * remove reverse ode comment * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * vae offload change * fix text encoder offloading * Apply style fixes * cleaner to_kwargs * fix module name in copied from * add requirements * fix offloading * fix offloading * fix offloading * update transformers version in reqs * try AutoTokenizer * try AutoTokenizer * Apply style fixes * empty commit * Delete tests/lora/test_lora_layers_hidream.py * change tokenizer_4 to load with AutoTokenizer as well * make text_encoder_four and tokenizer_four configurable * save model card * save model card * revert T5 * fix test * remove non diffusers lumina2 conversion --------- Co-authored-by: Bagheera <59658056+bghira@users.noreply.github.com> Co-authored-by: hlky <hlky@hlky.ac> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Small change requirements_sana.txt to requirements_hidream.txt
…e#11375) * fix * add tests * add message check
… for image resizing (huggingface#11395)
* Kolors additional pipelines, community contrib --------- Co-authored-by: Teriks <Teriks@users.noreply.github.com> Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com>
* 1. add pre-computation of prompt embeddings when custom prompts are used as well 2. save model card even if model is not pushed to hub 3. remove scheduler initialization from code example - not necessary anymore (it's now if the base model's config) 4. add skip_final_inference - to allow to run with validation, but skip the final loading of the pipeline with the lora weights to reduce memory reqs * pre encode validation prompt as well * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * pre encode validation prompt as well * Apply style fixes * empty commit * change default trained modules * empty commit * address comments + change encoding of validation prompt (before it was only pre-encoded if custom prompts are provided, but should be pre-encoded either way) * Apply style fixes * empty commit * fix validation_embeddings definition * fix final inference condition * fix pipeline deletion in last inference * Apply style fixes * empty commit * layers * remove readme remarks on only pre-computing when instance prompt is provided and change example to 3d icons * smol fix * empty commit --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Fix Flux IP adapter argument in the example IP-Adapter example had a wrong argument. Fix `true_cfg` -> `true_cfg_scale`
…for resizing (huggingface#11421) * Set LANCZOS as default interpolation mode for resizing * [train_dreambooth_lora.py] Set LANCZOS as default interpolation mode for resizing
…s in pipelines during torch.compile() (huggingface#11085) * test for better torch.compile stuff. * fixes * recompilation and graph break. * clear compilation cache. * change to modeling level test. * allow running compilation tests during nightlies.
* enable group_offload cases and quanto cases on XPU Signed-off-by: YAO Matrix <matrix.yao@intel.com> * use backend APIs Signed-off-by: Yao Matrix <matrix.yao@intel.com> * fix style Signed-off-by: Yao Matrix <matrix.yao@intel.com> --------- Signed-off-by: YAO Matrix <matrix.yao@intel.com> Signed-off-by: Yao Matrix <matrix.yao@intel.com>
* enable test_layerwise_casting_memory cases on XPU Signed-off-by: Yao Matrix <matrix.yao@intel.com> * fix style Signed-off-by: Yao Matrix <matrix.yao@intel.com> --------- Signed-off-by: Yao Matrix <matrix.yao@intel.com>
fix import.
…follow up (huggingface#11426) * Update train_text_to_image.py * update
…ipts follow up (huggingface#11427) * Update train_text_to_image_lora.py * update_train_text_to_image_lora
* enable gguf test cases on XPU Signed-off-by: YAO Matrix <matrix.yao@intel.com> * make SD35LargeGGUFSingleFileTests::test_pipeline_inference pas Signed-off-by: root <root@a4bf01945cfe.jf.intel.com> * make FluxControlLoRAGGUFTests::test_lora_loading pass Signed-off-by: Yao Matrix <matrix.yao@intel.com> * polish code Signed-off-by: Yao Matrix <matrix.yao@intel.com> * Apply style fixes --------- Signed-off-by: YAO Matrix <matrix.yao@intel.com> Signed-off-by: root <root@a4bf01945cfe.jf.intel.com> Signed-off-by: Yao Matrix <matrix.yao@intel.com> Co-authored-by: root <root@a4bf01945cfe.jf.intel.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
remove unnecessary pipeline moving to cpu in validation Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Fixing missing provider options argument * Adding if else for provider options * Apply suggestions from code review Co-authored-by: YiYi Xu <yixu310@gmail.com> * Apply style fixes * Update src/diffusers/pipelines/onnx_utils.py Co-authored-by: YiYi Xu <yixu310@gmail.com> * Update src/diffusers/pipelines/onnx_utils.py Co-authored-by: YiYi Xu <yixu310@gmail.com> --------- Co-authored-by: Uros Petkovic <urpektov@amd.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…lNet training (huggingface#11449) Set LANCZOS as the default interpolation for image resizing
…gingface#11425) raise warning instead of error
Signed-off-by: Yao Matrix <matrix.yao@intel.com>
Signed-off-by: Yao Matrix <matrix.yao@intel.com>
* enable unidiffuser cases on XPU Signed-off-by: Yao Matrix <matrix.yao@intel.com> * fix a typo Signed-off-by: Yao Matrix <matrix.yao@intel.com> * fix style Signed-off-by: Yao Matrix <matrix.yao@intel.com> --------- Signed-off-by: Yao Matrix <matrix.yao@intel.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds SEGA for FLUX, thereby enable users to guide the diffusion process by applying editing prompts.