Add support for optimizing clip models

[pakelley]

What does this PR do?

This allows optimization of clip and clip_text_model onnx models.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[pakelley]

I'm having trouble getting the tests to run, but it's not obvious what tests I would need to add? From what I can tell, there's nothing currently testing the clip_text_model that I might build off of.
The Adding support for an unsupported architecture mention adding your model to tests/exporters/test_onnx_export.py (which I figure should be tests/exporters/onnx/test_onnx_export.py), but I don't see anything obvious there where I'd add "clip" or "clip_text_model".
I also don't see the models available for optimization listed in the docs, but if there's such a place I'm happy to add these there.
Am I missing anything I'd need for this PR to be approved? Happy to work towards getting this up to standards, but from what I can tell there isn't much in the way of tests/docs required?

[tianleiwu]

These new arguments are for UNet, and not CLIP. Note that those operators only works in GPU and not CPU right now.

NhwcConv exists in ORT since 1.14. It is used to speed up convolution for GPU inference (new GPUs with tensor cores).

CLIP does not have convolution. These arguments have no impact on it.

[pakelley]

Oh, gotcha! I was getting errors without this, but I can test again without that. Are you saying this line should be removed?

[pakelley]

When I try to optimize a model without disabling NhwcConv, I get this error:

AttributeError                            Traceback (most recent call last)
Cell In[3], line 30
---> 30     optimizer.optimize(
     31         save_dir=output_dir, optimization_config=optimization_config, file_suffix=""
     32     )

File /path/to/optimum/optimum/onnxruntime/optimization.py:166, in ORTOptimizer.optimize(self, optimization_config, save_dir, file_suffix, use_external_data_format, one_external_file)
    163 maybe_save_preprocessors(self.onnx_model_path[0].parent, save_dir)
    165 model_type = ORTConfigManager.get_model_ort_type(self.config.model_type)
--> 166 optimization_options = optimization_config.create_fusion_options(model_type)
    168 logger.info("Optimizing model...")
    170 # TODO: this is quite inefficient as we load in memory if models are <2GB without external data

File /path/to/optimum/optimum/onnxruntime/configuration.py:825, in OptimizationConfig.create_fusion_options(self, model_type)
    822         continue
    823     setattr(args, attr, value)
--> 825 return FusionOptions.parse(args)

File /path/to/onnxruntime/transformers/fusion_options.py:105, in FusionOptions.parse(args)
    102     options.disable_attention_mask()
    104 if args.model_type in ["unet", "vae", "clip"]:
--> 105     if args.disable_nhwc_conv:
    106         options.enable_nhwc_conv = False
    107     if args.disable_group_norm:

AttributeError: 'Box' object has no attribute 'disable_nhwc_conv'

This is using onnxruntime version 1.15.1, for what it's worth.

[github-actions]

This PR has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.