Add TrOCR + VisionEncoderDecoderModel

[NielsRogge]

What does this PR do?

This PR adds the TrOCR models by Microsoft, together with a new VisionEncoderDecoderModel class (which should be used in order to use TrOCR, as it consists of an image encoder and an autoregressive text decoder). This PR is very similar to #13186, it's just the vision counterpart.

Here's how to use this model:

from transformers import TrOCRProcessor, VisionEncoderDecoderModel
import requests
from PIL import Image

processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")
model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")

url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
pixel_values = processor(image, return_tensors="pt").pixel_values

generated_ids = model.generate(input_ids=pixel_values)
print(processor.batch_decode(generated_ids)[0])

There's also this Colab notebook for quick inference: https://colab.research.google.com/drive/1qCuqlqc4V9LZhPkxIi_XqCCTQrDhkmHi?usp=sharing

! A big disclaimer: the TrOCR models do not directly work on entire images of PDFs etc. They are trained on single-text-line images. One needs a text detector first, before applying TrOCR. TrOCR is a text recognition model, not a text detection model. One typically needs both in a sequence in order to extract all text from a given image.

Important note:

The current design of the existing EncoderDecoderModel/FlaxEncoderDecoderModel is that, if the hidden_size of the encoder/decoder don't match, one creates a single projection layer to project the encoder_hidden_states to the same number of channels as the decoder. However, for TrOCR, this is not how it's done. Instead, one projects the encoder_hidden_states to the same dimension as the decoder when projecting to keys and values, in each decoder layer. Therefore, my proposal is to add an attribute to the config of the decoder called encoder_hidden_size, which, if specified, will be used in the VisionEncoderDecoderModel class to not project the encoder hidden states. Instead, it will be used when instantiating the key and value projection layers.

For consistency, we could also add this to the existing EncoderDecoderModel/FlaxEncoderDecoderModel. Also relevant for the FlaxVisionEncoderDecoderModel PR, see #13359.

To do:

• currently you have to pass input_ids=pixel_values to the generate method, which is not ideal. I'm in favor of letting the generate method accept an argument called inputs, which can work with text, images, speech. Related to Consistent speech model input names for the Seq2SeqTrainer generate function #13825.
• make sure all vision models accept attention_mask as input. Related to Add attention-mask support for ViTModel #13764.

[patrickvonplaten]

this isn't used anywhere no? Is it just here since attention_mask is often passed in generate()?

[patrickvonplaten]

PR looks great to me! Really clean implementation that fits well with the current design of Transformers IMO - it should enable lots of image captioning tasks from pretrained ViT + BERT :-)
Also very nice that you didn't have to adapt generate() to make it work!

Just have the following points:

• Do we need this attention_mask here: Add TrOCR + VisionEncoderDecoderModel #13874 (comment) ? I didn't dive to deep into the code, but if we need it just because generate(...) passes it then it's a bit hacky and we should try to avoid it. Happy to see how we can change generate for this or at least better add a kwargs(...) arguments that logs that the input is not used.
• I feel quite strongly about not calling it encoder_hidden_size, but rather cross_attention_hidden_size. From a user that just looks at configuration_utils.py the name encoder_hidden-size is not at all related to encoder-decoder architectures. Can we change that maybe?
• Can we add one slow integration test with a real model (maybe the one in your notebook?)

Overall amazing work! Think we can merge this in a couple of days :-)

[sgugger]

Thanks a lot for adding this!

[sgugger]

This comment should be expanded or removed.

[LysandreJik]

This looks great. Thanks for working on this, @NielsRogge!

[LysandreJik]

How does it differ?

[NielsRogge]

@patrickvonplaten knows, I just copied his implementation

[NielsRogge]

The same docstring is used in Speech2Text2