[WIP] Add MMS and Wav2Vec2 models (Closes #209)

[xenova]

TODO:

• Feature extraction w/ wav2vec2
• Automatic Speech Recognition w/ wav2vec2
• Audio classification w/ wav2vec2
• Feature extraction w/ MMS
• Audio classification w/ MMS
• Automatic Speech Recognition w/ MMS (python implementation) NOTE Due to limitations of ONNX for exporting models with adapters, this functionality will be moved to a future release.
  • https://huggingface.co/patrickvonplaten/mms-1b-fl102
  • https://huggingface.co/patrickvonplaten/mms-1b-l1107
  • https://huggingface.co/patrickvonplaten/mms-1b-all
• Text-to-speech w/ MMS (python implementation still in progress) Moved to a future release

Usage

Feature extraction

import { AutoProcessor, read_audio } from '@xenova/transformers';

// Read and preprocess audio
const processor = await AutoProcessor.from_pretrained('Xenova/mms-300m');
const audio = await read_audio('https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac', 16000);
const inputs = await processor(audio);

// Run model with inputs
const model = await AutoModel.from_pretrained('Xenova/mms-300m');
const output = await model(inputs);
// {
//   last_hidden_state: Tensor {
//     dims: [ 1, 1144, 1024 ],
//     type: 'float32',
//     data: Float32Array(1171456) [ ... ],
//     size: 1171456
//   }
// }

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.