still low performances with CoreML model (using ANE) compared to Vosk (Kaldi) using CPU

[xcottos]

Hi,

using the large CoreML encoder provided by huggingface I still have performances very low compared to Vosk with Kaldi and I don't get why.

when I run it:

whisper_init_state: Core ML model loaded
system_info: n_threads = 4 / 10 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | COREML = 1 | OPENVINO = 0 |

so everything is finely set and in theory I should use ANE that should be really performant.

for converting 3h audio (16khz and 1channel) it took 1h, while with Vosk using Kaldi (I use only the CPU), same quality but it took 11min, how is it possible? Am I missing something?

Thank you
Luca

[bobqianic]

I still have performances very low compared to Vosk with Kaldi and I don't get why.

Compared to the Whisper large model, which is 2.9 GiB, the Vosk and Kaldi models are relatively small. In addition, the KAIDI GigaSpeech ASR models were trained on datasets that are an order of magnitude smaller than Whisper (10,000 hours vs 680,000 hours). If you're aiming for faster transcription speeds, you might want to consider using Metal instead of CoreML or other smaller models (eg. small, medium).

In this work we close that gap, scaling weakly supervised speech recognition the next order of magnitude to 680,000 hours of labeled audio data. We call our approach Whisper.

[xcottos]

Thank you bobqianic,

I can see, indeed, that the transcription with whisper is much accurate (writes numbers as numbers, the punctuation is much better etc) so I assume the better accuracy is also the result of a larger training set/model.

I am studying the options I have and I would like to push whisper at its max performances on my M1 Max (64GB Ram and 32 cores GPU). Now I started studying the acceleration with my Mac and as far as I understood ANE should be even more performant compared to Metal but it's still only theory for me that's why I'm experimenting.

For using ANE the only way (as far as I could get) is using CoreML and apparently I am using it now (so in theory I am using the NPU)

Is there already the support for Metal with whisper to test its performances on my GPU?

Thank you
Luca

[bobqianic]

Is there already the support for Metal with whisper to test its performances on my GPU?

Yes. See #1270

[xcottos]

well... great improvement it took 20 min now with Metal :)

large model and 3h audio in Italian

I still don't get why ANE is less performant...

Thanks for your suggestion bobqianic, really appreciate it

Luca