[MMS] Unable to load processor for `facebook/mms-300m` or `facebook/mms-1b`

[xenova]

System Info

• transformers version: 4.32.0.dev0
• Platform: Linux-5.15.109+-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.16.4
• Safetensors version: 0.3.2
• Accelerate version: not installed
• Accelerate config: not found
• PyTorch version (GPU?): 2.0.1+cu118 (False)
• Tensorflow version (GPU?): 2.12.0 (False)
• Flax version (CPU?/GPU?/TPU?): 0.7.2 (cpu)
• Jax version: 0.4.14
• JaxLib version: 0.4.14
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help?

@sanchit-gandhi

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Running

from transformers import AutoProcessor
AutoProcessor.from_pretrained('facebook/mms-300m')

Produces:

/usr/local/lib/python3.10/dist-packages/transformers/models/wav2vec2/processing_wav2vec2.py:53: FutureWarning: Loading a tokenizer inside Wav2Vec2Processor from a config that does not include a `tokenizer_class` attribute is deprecated and will be removed in v5. Please add `'tokenizer_class': 'Wav2Vec2CTCTokenizer'` attribute to either your `config.json` or `tokenizer_config.json` file to suppress this warning: 
  warnings.warn(
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/transformers/models/wav2vec2/processing_wav2vec2.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
     50         try:
---> 51             return super().from_pretrained(pretrained_model_name_or_path, **kwargs)
     52         except OSError:

7 frames
OSError: Can't load tokenizer for 'facebook/mms-300m'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'facebook/mms-300m' is the correct path to a directory containing all relevant files for a Wav2Vec2CTCTokenizer tokenizer.

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, *init_inputs, **kwargs)
   1828 
   1829         if all(full_file_name is None for full_file_name in resolved_vocab_files.values()):
-> 1830             raise EnvironmentError(
   1831                 f"Can't load tokenizer for '{pretrained_model_name_or_path}'. If you were trying to load it from "
   1832                 "'https://huggingface.co/models', make sure you don't have a local directory with the same name. "

OSError: Can't load tokenizer for 'facebook/mms-300m'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'facebook/mms-300m' is the correct path to a directory containing all relevant files for a Wav2Vec2CTCTokenizer tokenizer.

Expected behavior

The correct processor should be loaded (Wav2Vec2FeatureExtractor from the preprocessor_config.json).
The error message suggests the tokenizer is mandatory for all MMS models, which isn't necessarily the case (specifically for just loading the pretrained base models).

[sanchit-gandhi]

The processor class is made up of two objects:

1. The feature extractor: to pre-process the audio inputs
2. The tokenizer: to convert the predicted CTC ids to text outputs

Note that the tokenizer is only required for CTC decoding Wav2Vec2ForCTC. If you only require the base model (Wav2Vec2Model), then you can get away with just using the feature extractor to pre-process the inputs:

from transformers import AutoFeatureExtractor

model_id = "facebook/mms-300m"
feature_extractor = AutoFeatureExtractor.from_pretrained('facebook/mms-300m')

Each language in MMS has a different vocabulary, and hence a different tokenizer. Thus, we need to specify the target language in order to know the correct tokenizer to load.

Can you try setting the target language to load the target tokenizer? E.g. as follows:

from transformers import AutoProcessor

model_id = "facebook/mms-1b-all"
target_lang = "fra"

processor = AutoProcessor.from_pretrained(model_id, target_lang=target_lang)

This will load the tokenizer for the French language under the hood.

[xenova]

Thanks! I was aware of AutoFeatureExtractor.from_pretrained, but I assumed that AutoProcessor would handle the case where a tokenizer is not available for the model type (i.e., for pretraining or audio classification). I believe my misunderstanding stemmed from other Processor classes (like those for image classification) which do not need tokenizers, e.g.,:

import requests
import torch
from PIL import Image
from transformers import AutoProcessor, MobileViTV2ForSemanticSegmentation

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoProcessor.from_pretrained("apple/mobilevitv2-1.0-imagenet1k-256")
model = MobileViTV2ForSemanticSegmentation.from_pretrained("apple/mobilevitv2-1.0-imagenet1k-256")

inputs = image_processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

# logits are of shape (batch_size, num_labels, height, width)
logits = outputs.logits
print(logits.shape) # torch.Size([1, 1000, 8, 8])

Anyway, thanks for the response - I'll close the issue.

[sgugger]

@xenova MobileVit does not have a processor class, so AutoProcessor.from_pretrained returns to you a simple ImageProcessor.

AutoProcessor.from_pretrained will always return the tool for preprocessing starting with Processor, then Tokenizer, then FeatureExtractor and finally ImageProcessor. For your example above, there is a processor class, so it tried to load this one, but for MobileViT, there is none, so it automatically goes down to the ImageProcessor. Does that make sense?

[xenova]

@sgugger Great! Thanks for the explanation! 🤗