MusicGen fails when being used with pipeline

[osanseviero]

System Info

• transformers version: 4.40.2
• Platform: Linux-6.1.85+-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.23.1
• Safetensors version: 0.4.3
• Accelerate version: not installed
• Accelerate config: not found
• PyTorch version (GPU?): 2.3.0+cu121 (True)
• Tensorflow version (GPU?): 2.15.0 (True)
• Flax version (CPU?/GPU?/TPU?): 0.8.3 (gpu)
• Jax version: 0.4.26
• JaxLib version: 0.4.26
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@sanchit-gandhi @ylacombe

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

This code gives an error

from transformers import pipeline

pipe = pipeline("text-to-audio", model="facebook/musicgen-small", device="cuda")
data = pipe("electric rock solo, very intense")

RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

Colab with different versions: https://colab.research.google.com/drive/1qYDk9_rbs7lC2EfqL4q-J9weMFyO7F7X#scrollTo=fzZBH7O4xjSQ

Expected behavior

It should work just as it did in older versions (e.g. 4.39.2 works)

Seems related to #30020

[ylacombe]

Thanks for opening the issue @osanseviero, I've been digging this up a bit and I believe I finally got the reason why it and #30020 happened.

The reason is that SDPA produces Nan when given a padding mask that attends to no position at all (see this thread).
In the case of Musicgen, it can happen for two reasons:

1. when we use get_unconditional_inputs (here)
2. when guidance_scale>1 (see here)

Still have to think how to solve this but we'll probably switch back to eager attention when the padding mask falls into this edge case! (it happens only in the cross-attention layer btw)