[Feature request] Add support for Massively Multilingual Speech(MMS) model

[bil-ash]

Please consider adding support for the asr and tts features of the MMS model to transformers.js.

The model docs are available here . The pytorch ASR model finetuned for 102 languages is available [here]https://huggingface.co/facebook/mms-1b-fl102/blob/main/pytorch_model.bin) . It(3.86GB) is not much larger than the whisper-medium(3.06GB) but works much better on Indian languages like Hindi, Bengali and Assamese(not benchmarked, I know the languages and so analysed the results obtained for a few audio clips in these languages).

Also, the TTS model can be a good contender for the TTS pipeline for transformers about which there is an ongoing work in transformers. Apparently, this is the transformers TTS model with the largest number of supported languages.

[xenova]

Thanks for the suggestion, I will look into it. In the meantime, could you provide example code for how you would use it in python (as well as the expected output)?

[bil-ash]

The necessary docs are available at https://huggingface.co/docs/transformers/main/en/model_doc/mms

[sanchit-gandhi]

For reference, the MMS ASR model is integrated into transformers (huggingface/transformers#23813). The MMS TTS integration is on-going, but should be finished this week: huggingface/transformers#24085.

[xenova]

Work in progress: #220

[xenova]

I have added MMS for feature-extraction and audio classification, however, the speech recognition is slightly more complicated. From what I can tell, the python library allows users to update so-called adapter weights for the model during runtime. Unfortunately, this is not possible with ONNX models (afaik).

@fxmarty @michaelbenayoun any ideas? 😅

[fxmarty]

@xenova yes I had a similar issue implementing ControlNet ONNX export - basically some models having a base + a tiny extension (like LoRA weights, which is a simple case because they can be fused). huggingface/optimum#970 The ONNX format does not support that (easily) unfortunately, you would need to export all at once.