enable static cache on vision encoder decoder

[jiqing-feng]

After the issue #39746 fixed, the vision encoder decoder model can support static cache and can got more significant speed-up

import time
import requests
import torch
import PIL.Image
from transformers import pipeline

model_id = "nlpconnect/vit-gpt2-image-captioning"
image_to_text = pipeline("image-to-text", model=model_id, device="cpu", torch_dtype=torch.float16)
image_url = "https://ankur3107.github.io/assets/images/image-captioning-example.png"
image = PIL.Image.open(requests.get(image_url, stream=True, timeout=3000).raw)
generation_config = image_to_text.model.generation_config
generation_config.cache_implementation = "static"

for _ in range(10):
    output = image_to_text(image, generate_kwargs={"generation_config": generation_config})

start = time.time()
output = image_to_text(image, generate_kwargs={"generation_config": generation_config})
end = time.time()
print(f"eager mode pipeline latency {end - start}")

image_to_text.model.forward = torch.compile(image_to_text.model.forward)

for _ in range(10):
    output = image_to_text(image, generate_kwargs={"generation_config": generation_config})

start = time.time()
output = image_to_text(image, generate_kwargs={"generation_config": generation_config})
end = time.time()
print(f"compile mode pipeline latency {end - start}")

[jiqing-feng]

Hi @zucchini-nlp , could you please review this change? Thanks!

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: vision_encoder_decoder

[jiqing-feng]

run-slow: vision_encoder_decoder

[zucchini-nlp]

Yeah, makes sense since the model can support compile as long as the LM supports it. Usually we need these to be class attributes, so we can check before init the class. For vision encoder decoder though we can't do it yet, so it's fine

Thanks!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.