Diffusers support #604

quic-amitraj · 2025-11-04T06:09:40Z

Support for Diffusers Architecture in Efficient Transformers

Overview

This pull request introduces Diffusers architecture support to the Efficient Transformers framework, enabling seamless integration of diffusion models.

Key Highlights

Support of model black-forest-labs/FLUX1-schnell
Flexible Configuration
- Supports JSON-based configuration files for easy compilation and execution.
Performance Benchmarking
- Implements a performance matrix for Diffusers models to enable benchmarking for each modules.
Testing Framework
- Includes initial test scripts for Diffusers (In progress).
Support of onnx subfunction graph using flag use_onnx_function
Support parallel compilation of modules using flag parallel_compile

scripts/Jenkinsfile

examples/diffusers/flux/flux_1_schnell.py

examples/diffusers/flux/flux_1_shnell_custom.py

ochougul · 2025-11-20T09:40:26Z

examples/diffusers/flux/flux_1_shnell_custom.py

+#     text_encoder=custom_text_encoder,      # Your custom CLIP text encoder
+#     transformer=custom_transformer,         # Your custom transformer model
+#     tokenizer=custom_tokenizer,             # Your custom tokenizer


this is nice.

QEfficient/__init__.py

Signed-off-by: Amit Raj <amitraj@qti.qualcomm.com> Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

…nd comments addressed Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

… export 3. Hash getting changed after each export Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

QEfficient/diffusers/pipelines/flux/pipeline_flux.py

QEfficient/diffusers/models/autoencoders/autoencoder_kl.py

QEfficient/diffusers/models/transformers/transformer_flux.py

QEfficient/diffusers/pipelines/flux/pipeline_flux.py

quic-rishinr · 2025-11-27T08:10:52Z

QEfficient/diffusers/pipelines/flux/pipeline_flux.py

+    def encode_prompt(
+        self,
+        prompt: Union[str, List[str]],
+        prompt_2: Optional[Union[str, List[str]]] = None,


nit(suggestion): better to use prompt and secondary_prompt as variable names

These are official names from diffusers, as we are alligned with diffusers pipeline, In my sugggestion we should keep the name same. Plaease refer this- Link

QEfficient/diffusers/pipelines/flux/pipeline_flux.py

QEfficient/diffusers/pipelines/pipeline_module.py

quic-rishinr · 2025-11-28T05:30:51Z

QEfficient/diffusers/README.md

@@ -0,0 +1,95 @@
+
+<div align="center">


move it to docs folder and create a new page for diffusers

Sure Rishin, Will put this in the docs as well as it will be helpful for the sphinx documentation. But keeping it here also make sense as it gives the proper idea about `How anyone can run diffusers supported model using Qefficient?"

quic-rishinr · 2025-11-28T06:18:34Z

examples/diffusers/flux/flux_1_schnell.py

@@ -0,0 +1,55 @@
+# -----------------------------------------------------------------------------


Add a readme file explaining the usage, follow the the contribution.md file in the example folder

quic-rishinr · 2025-11-28T06:19:32Z

scripts/Jenkinsfile

           parallel {
               stage('Run Non-CLI Non-QAIC Tests') {
                   steps {
-                       timeout(time: 25, unit: 'MINUTES') {


Create a new tag and add tests as well

Yes, with the following PR we will add pytest as well.

Added here- PR to add pytest

Please review.

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

QEfficient/utils/constants.py

QEfficient/utils/_utils.py

QEfficient/transformers/models/t5/modeling_t5.py

ochougul · 2025-12-02T15:18:03Z

QEfficient/utils/export_utils.py

+        try:
+            # 4. Execute the actual export
+            onnx_path = func(self, *args, **kwargs)
+
+            # 5. Save export metadata
+            _save_export_metadata(export_dir, filtered_hash_params)
+
+            return onnx_path
+
+        finally:
+            # 6. Always cleanup subfunctions if they were setup
+            if use_subfunctions:
+                _cleanup_onnx_subfunctions(self)


try/finally is not required. Let's not loop one try/except inside another.

QEfficient/utils/hash_utils.py

QEfficient/diffusers/models/transformers/transformer_flux.py

QEfficient/diffusers/models/pytorch_transforms.py

QEfficient/diffusers/pipelines/flux/flux_config.json

QEfficient/diffusers/pipelines/flux/pipeline_flux.py

ochougul · 2025-12-02T17:39:26Z

QEfficient/diffusers/pipelines/flux/pipeline_flux.py

+            **kwargs,
+        )
+
+    def export(self, export_dir: Optional[str] = None, use_onnx_subfunctions: bool = False) -> str:


is it possible to make this logic common for any diffusers model?
User only defines an ordered dictionary and uses the export method from base class?
user's job is to define a orderdict for modules and the get_onnx_config method for export params.

With logic of having on more base class, this could be possible. In upcoming time will add support of QEffDiffusionPipeline as a base class, which will have all these logic common.

ochougul · 2025-12-02T17:39:46Z

QEfficient/diffusers/pipelines/flux/pipeline_flux.py

+        """
+        for module_name, module_obj in tqdm(self.modules.items(), desc="Exporting modules", unit="module"):
+            # Get ONNX export configuration for this module
+            example_inputs, dynamic_axes, output_names = module_obj.get_onnx_config()


name the method better you are not returning a config here.

QEfficient/diffusers/pipelines/flux/pipeline_flux.py

ochougul · 2025-12-02T17:41:22Z

QEfficient/diffusers/pipelines/flux/pipeline_flux.py

+            )
+
+        # Initialize QAIC inference session if not already created
+        if self.text_encoder_2.qpc_session is None:


why are we keeping session in memory?
This will hold the card occupied unless we explicitly del this or program execution terminates

We want to keep it in the memory for genertating multple images sequentially in loop and not load the qpc again and again after each image generation.

QEfficient/diffusers/pipelines/flux/pipeline_flux.py

ochougul · 2025-12-02T17:48:53Z

QEfficient/diffusers/pipelines/pipeline_utils.py

+from QEfficient.utils.logging_utils import logger
+
+
+def calculate_compressed_latent_dimension(height: int, width: int, vae_scale_factor: int) -> int:


is this method common to all diffusion models?

QEfficient/transformers/models/t5/modeling_t5.py

quic-amitraj changed the title ~~Flux support with custom config~~ Diffusers support Nov 4, 2025

quic-amitraj force-pushed the config_support_diffusers branch 5 times, most recently from ffed450 to eb55c51 Compare November 13, 2025 08:47

quic-amitraj marked this pull request as ready for review November 13, 2025 08:48

quic-amitraj requested review from ochougul, quic-hemagnih and quic-rishinr as code owners November 13, 2025 08:49

quic-amitraj force-pushed the config_support_diffusers branch 2 times, most recently from 42cf8b6 to 1cf4b30 Compare November 13, 2025 09:07

quic-amitraj marked this pull request as draft November 13, 2025 09:25

quic-amitraj force-pushed the config_support_diffusers branch from 1cf4b30 to 0ee0b9f Compare November 13, 2025 09:29

quic-amitraj marked this pull request as ready for review November 13, 2025 09:30

quic-amitraj force-pushed the config_support_diffusers branch from 0ee0b9f to 1e0fbfd Compare November 13, 2025 10:08

quic-amitraj requested a review from anujgupt-github November 13, 2025 12:59

quic-amitraj self-assigned this Nov 13, 2025

quic-amitraj force-pushed the config_support_diffusers branch 2 times, most recently from bb01d84 to 00f5286 Compare November 13, 2025 13:24

quic-amitraj added the Diffusers Use for PR related to diffusers in efficient-transformers. label Nov 13, 2025

quic-amitraj force-pushed the config_support_diffusers branch from 00f5286 to 0a26f1b Compare November 13, 2025 16:52

quic-amitraj requested a review from vbaddi November 14, 2025 04:40

quic-amitraj force-pushed the config_support_diffusers branch from ac6382a to 19e0ccd Compare November 14, 2025 10:24

ochougul reviewed Nov 20, 2025

View reviewed changes

QEfficient/__init__.py Outdated Show resolved Hide resolved

quic-amitraj force-pushed the config_support_diffusers branch from 19e0ccd to 8d78ac9 Compare November 24, 2025 08:29

quic-amitraj added 3 commits November 27, 2025 09:36

Stage-1 export debug

9d41cd6

Signed-off-by: Amit Raj <amitraj@qti.qualcomm.com> Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Stage-2 Export inital working version done

949d86b

Signed-off-by: Amit Raj <amitraj@qti.qualcomm.com> Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Stage-3 compilation work is under progress

4aae90f

Signed-off-by: Amit Raj <amitraj@qti.qualcomm.com> Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Amit Raj added 9 commits November 27, 2025 09:36

Added tqdm for export and compile

1b08fa9

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Parallel compilation and onnx subfunction is added

02c1c05

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Height and widht now can be passed from compile and __call__ method a…

c10ee5a

…nd comments addressed Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Removed redundant code

424fc83

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Minor fixes-1

31fd747

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

addressed comments and fixes

b2f8668

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Handled 1. Mutiple time export issue 2. Meta device error after first…

30a2c46

… export 3. Hash getting changed after each export Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Code cleanup and fixes

cfe61b9

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Updated readme

901d293

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

quic-amitraj force-pushed the config_support_diffusers branch from 226e65a to 901d293 Compare November 27, 2025 09:37

quic-rishinr requested changes Nov 27, 2025

View reviewed changes

quic-rishinr requested changes Nov 28, 2025

View reviewed changes

Updated the class names

671f53c

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

quic-amitraj force-pushed the config_support_diffusers branch from fc06ca3 to 901d293 Compare December 2, 2025 05:28

Amit Raj added 2 commits December 2, 2025 08:28

Addressed comments-2

5a74629

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

Addressed comments-3

2d2fc38

Signed-off-by: Amit Raj <amitraj@qti.qualcommm.com>

ochougul reviewed Dec 2, 2025

View reviewed changes

QEfficient/diffusers/models/pytorch_transforms.py Show resolved Hide resolved

ochougul reviewed Dec 2, 2025

View reviewed changes

QEfficient/diffusers/pipelines/flux/flux_config.json Show resolved Hide resolved

ochougul reviewed Dec 2, 2025

View reviewed changes

QEfficient/diffusers/pipelines/flux/pipeline_flux.py Show resolved Hide resolved

ochougul reviewed Dec 2, 2025

View reviewed changes

QEfficient/diffusers/pipelines/flux/pipeline_flux.py Show resolved Hide resolved

ochougul reviewed Dec 2, 2025

View reviewed changes

QEfficient/diffusers/pipelines/flux/pipeline_flux.py Show resolved Hide resolved

ochougul reviewed Dec 2, 2025

View reviewed changes

QEfficient/diffusers/pipelines/flux/pipeline_flux.py Show resolved Hide resolved

ochougul reviewed Dec 2, 2025

View reviewed changes

QEfficient/diffusers/pipelines/flux/pipeline_flux.py Show resolved Hide resolved

ochougul reviewed Dec 2, 2025

View reviewed changes

QEfficient/transformers/models/t5/modeling_t5.py Show resolved Hide resolved

ochougul reviewed Dec 2, 2025

View reviewed changes

QEfficient/transformers/models/t5/modeling_t5.py Show resolved Hide resolved

		@@ -0,0 +1,55 @@
		# -----------------------------------------------------------------------------

		from QEfficient.utils.logging_utils import logger


		def calculate_compressed_latent_dimension(height: int, width: int, vae_scale_factor: int) -> int:

Diffusers support #604

Are you sure you want to change the base?

Diffusers support #604

Conversation

quic-amitraj commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Support for Diffusers Architecture in Efficient Transformers

Overview

Key Highlights

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quic-amitraj Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

quic-amitraj commented Nov 4, 2025 •

edited

Loading

quic-amitraj Nov 28, 2025 •

edited

Loading