Add vision encoder decoder model in exporters #588

mht-sharma · 2022-12-14T06:58:24Z

What does this PR do?

Supports vision-encoder-decoder export to onnx in optimum

Changes

Adds generic class for exporting Encoder-Decoder type models doc
Adds export for vision-encoder-decoder models using the above class

Limitation

Past key value export not supported with TROcr
Donut model is not supported

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

HuggingFaceDocBuilderDev · 2022-12-14T07:16:01Z

The documentation is not available anymore as the PR was closed or merged.

fxmarty · 2023-01-23T13:06:53Z

tests/exporters/exporters_utils.py

@@ -110,6 +110,7 @@
    "speech-to-text": "hf-internal-testing/tiny-random-Speech2TextModel",
    "xlm": "hf-internal-testing/tiny-random-XLMModel",
    "xlm-roberta": "hf-internal-testing/tiny-xlm-roberta",
+    "vision-encoder-decoder": "hf-internal-testing/tiny-random-VisionEncoderDecoderModel-vit-gpt2",


Could you add a test for trocr and donut-swin as well?

added for trocr, donut-skin has threshold issue while exporting would look into it separately

Then could we not add it in tasks.py? And handle adding donut-swin in a separate PR altogether?

The test model has high atol requirement (4e-2). The export for the model in doc is slow but works with 1e-3 threshold, so not putting that here.

What do you mean by the model in doc?

I'd just like that what is said to be supported in tasks.py is actually supported, and we can make sure of it by adding an unit test.

Commented Donut. Will handle this separately

fxmarty · 2023-01-23T13:11:04Z

optimum/exporters/onnx/config.py

+            behavior=behavior,
+        )
+
+        from ..tasks import TasksManager


would we have it at the top level?

It causes circular export on top

This could mean that some refactor is needed, I guess.

Deferring import would still be ok IMO. But if there is a better way would look into it.

fxmarty · 2023-01-23T13:14:07Z

optimum/utils/normalized_config.py

+    def __getattr__(self, attr_name):
+        if (
+            self.ENCODER_CONFIG is not None
+            and self.ENCODER_NORMALIZED_CONFIG_CLASS is not None
+            and attr_name.upper() in dir(self.ENCODER_NORMALIZED_CONFIG_CLASS)
+        ):
+            return self.ENCODER_NORMALIZED_CONFIG_CLASS.__getattr__(attr_name)
+        if (
+            self.DECODER_CONFIG is not None
+            and self.DECODER_NORMALIZED_CONFIG_CLASS is not None
+            and attr_name.upper() in dir(self.DECODER_NORMALIZED_CONFIG_CLASS)
+        ):
+            return self.DECODER_NORMALIZED_CONFIG_CLASS.__getattr__(attr_name)
+
+        return super().__getattr__(attr_name)


why do we need this?

These type of models can have any decoder and encoder head and it is difficult to manage every normalised class. So created this helper class to work with these models

optimum/exporters/onnx/model_configs.py

optimum/utils/normalized_config.py

michaelbenayoun · 2023-01-24T10:13:10Z

optimum/exporters/onnx/config.py

+                    DummySeq2SeqDecoderTextInputGenerator,
+                    DummyPastKeyValuesGenerator,
+                )
+


I would add the nromalized config creation here instead of setting attributes in the if branches. I would also add a type check on it.

Is there a reason to set the values in constructor than setting attribute? Plus why would type check be necessary?

optimum/exporters/onnx/config.py

Co-authored-by: fxmarty <9808326+fxmarty@users.noreply.github.com>

fxmarty · 2023-01-31T09:14:08Z

optimum/utils/normalized_config.py

+class NormalizedEncoderDecoderConfig(NormalizedConfig):
+    ENCODER_NORMALIZED_CONFIG_CLASS = None
+    DECODER_NORMALIZED_CONFIG_CLASS = None
+
+    def __getattr__(self, attr_name):
+        if self.ENCODER_NORMALIZED_CONFIG_CLASS is not None and attr_name.upper() in dir(
+            self.ENCODER_NORMALIZED_CONFIG_CLASS
+        ):
+            return self.ENCODER_NORMALIZED_CONFIG_CLASS.__getattr__(attr_name)
+        if self.DECODER_NORMALIZED_CONFIG_CLASS is not None and attr_name.upper() in dir(
+            self.DECODER_NORMALIZED_CONFIG_CLASS
+        ):
+            return self.DECODER_NORMALIZED_CONFIG_CLASS.__getattr__(attr_name)
+
+        return super().__getattr__(attr_name)


Couldn't the attr_name be in both the encoder and decoder config? e.g. bos_token_id in https://huggingface.co/microsoft/trocr-small-handwritten/blob/main/config.json . Could this result in unexpected behavior? We should rather be able to access attributes recursively no, like normalized_config.encoder.bos_token_id or normalized_config.decoder.bos_token_id, if this makes sense!

Yes, it could be possible. This seems to be a common problem for the other hybrid configs also. Would make a note of it and handle it separately.
Having something like this normalized_config.encoder.bos_token_id or normalized_config.decoder.bos_token_id would require change in the entire input_generator so would need to think of a better way.
Currently, this would not affect the ORT export (where we export the encoder and decoder separately) as one of the configs would always be None. So considering it low priority.

fxmarty

LGTM, thanks!

Mir-Umar · 2023-04-10T10:43:15Z

Hi @mht-sharma can you kindly provide some insights about how to remove the following limitation from the above issue:

Past key value export not supported with TROcr
I am actually working with a custom trained TROCR model and want to convert it to ONNX for faster inference.

fxmarty · 2023-04-11T09:22:31Z

@Mir-Umar Feel free to open an issue as well.

mht-sharma · 2023-04-11T10:59:02Z

@Mir-Umar There is an existing issue for this #744. I did not got the time to work on this yet.

The use_cache argument for the TrOCR model does not work in Transformers so the export was failing. Probably the TROcr code in Transformers needs to be looked at to debug the issue further. Let me know if you would like to work on the above issue.

mht-sharma mentioned this pull request Dec 14, 2022

ONNX encoder decoder exchange invoke issue huggingface/transformers#20644

Closed

4 tasks

mht-sharma mentioned this pull request Dec 20, 2022

Add support for encoder-decoder models #367

Closed

mht-sharma mentioned this pull request Jan 2, 2023

Add onnx support for VisionEncoderDecoder huggingface/transformers#19254

Merged

mht-sharma force-pushed the add-support-vision-encoder-decoder-models branch from d6fef2f to 7e5932e Compare January 23, 2023 12:17

mht-sharma changed the title ~~Add support vision encoder decoder models~~ Add vision encoder decoder model in exporters Jan 23, 2023

mht-sharma requested review from michaelbenayoun and fxmarty January 23, 2023 12:32

mht-sharma marked this pull request as ready for review January 23, 2023 12:35

mht-sharma force-pushed the add-support-vision-encoder-decoder-models branch from d333eb4 to 7f3b18f Compare January 23, 2023 12:45

fxmarty reviewed Jan 23, 2023

View reviewed changes

optimum/exporters/onnx/model_configs.py Outdated Show resolved Hide resolved

michaelbenayoun reviewed Jan 24, 2023

View reviewed changes

mht-sharma and others added 13 commits January 30, 2023 12:01

updated configs

77b72bd

fix error

5ad4b88

fix errors

fa92f7d

add tests

0b5937d

remove duplicate code

065de70

fixed trocr

2e9e141

updated tests

baab7b9

Update optimum/exporters/onnx/model_configs.py

db4bec9

Co-authored-by: fxmarty <9808326+fxmarty@users.noreply.github.com>

fix cli tests

19c92c0

apply suggestions

1917c7a

improved norm config

ee3cd13

removed donut

b1b7df1

update the unsupported model with trocr

9b32631

mht-sharma force-pushed the add-support-vision-encoder-decoder-models branch from d949251 to 9b32631 Compare January 30, 2023 11:01

fxmarty reviewed Jan 31, 2023

View reviewed changes

fxmarty approved these changes Jan 31, 2023

View reviewed changes

michaelbenayoun approved these changes Jan 31, 2023

View reviewed changes

mht-sharma merged commit b2bba71 into huggingface:main Jan 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add vision encoder decoder model in exporters #588

Add vision encoder decoder model in exporters #588

mht-sharma commented Dec 14, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Dec 14, 2022 •

edited

Loading

fxmarty Jan 23, 2023

mht-sharma Jan 23, 2023

fxmarty Jan 23, 2023

mht-sharma Jan 23, 2023

fxmarty Jan 24, 2023

fxmarty Jan 24, 2023

mht-sharma Jan 24, 2023

fxmarty Jan 23, 2023

mht-sharma Jan 23, 2023

fxmarty Jan 23, 2023

mht-sharma Jan 23, 2023

fxmarty Jan 23, 2023

mht-sharma Jan 23, 2023

michaelbenayoun Jan 24, 2023 •

edited

Loading

mht-sharma Jan 24, 2023

fxmarty Jan 31, 2023

mht-sharma Jan 31, 2023 •

edited

Loading

fxmarty left a comment

Mir-Umar commented Apr 10, 2023

fxmarty commented Apr 11, 2023

mht-sharma commented Apr 11, 2023

Add vision encoder decoder model in exporters #588

Add vision encoder decoder model in exporters #588

Conversation

mht-sharma commented Dec 14, 2022 • edited Loading

What does this PR do?

Before submitting

HuggingFaceDocBuilderDev commented Dec 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelbenayoun Jan 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mht-sharma Jan 31, 2023 • edited Loading

Choose a reason for hiding this comment

fxmarty left a comment

Choose a reason for hiding this comment

Mir-Umar commented Apr 10, 2023

fxmarty commented Apr 11, 2023

mht-sharma commented Apr 11, 2023

mht-sharma commented Dec 14, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Dec 14, 2022 •

edited

Loading

michaelbenayoun Jan 24, 2023 •

edited

Loading

mht-sharma Jan 31, 2023 •

edited

Loading