CoreML conversion and model execution takes 11 hours

[EsbernTK]

After compiling this library with WHISPER_COREML=1 set, i tried converting the large model using the included script, however it takes 11 hours to compile, and then another 11 hours on the first run. I have encountered this issue before with smaller coreml models, and the solution there was to upgrade the coremltools to 7.0b2, which i did here too. However that hasnt worked, on a fresh install it is still 11 hours for compile and initial run.
The reason i suspect this is an issue with this repo and not coremltools itself, is that the coreml-encoder-large.mlpackage only takes 2 minutes to compile, while the ggml-encoder-large.mlmodelc takes the 11 hours and ggml-encoder-tiny.mlmodelc takes under a minute.
So if it was the same coremltools issue i encountered before, all three compilations would have taken an extremely long time, regardless of model size, but in this case it is only the ggml-encoder-large.mlpackage which seems to have an issue.

Specifications:
OS: MacOS Ventura
CPU: ARM64 M1
RAM: 16GB
Version: whisper.cpp latest commit
Python: ARM 3.9.7 and Universal 3.9.6
Coremltools: 7.0b2 and 6.3.0

[bobqianic]

Unfortunately, there's not much we can do about it. Please refer to #1278 for more details.

[bobqianic]

#773 #911 #937

[bobqianic]

You might want to give the latest Metal implementation a shot. It's not only faster and more robust, but it also ensures you won't run into such issues. See #1270

[EsbernTK]

Okay that is cool, how can i compile with Metal support? is it just using
make -j
or, i can see in the makefile that it looks for GGML_USE_METAL, so should i set that as an environment variable before compiling?

[bobqianic]

Okay that is cool, how can i compile with Metal support?

Yes, you can just use make -j. On Apple machines, Metal is set to ON by default.

https://github.com/ggerganov/whisper.cpp/blob/7e1592d2cdb3e6aa02faeaa7c74a1f39f2bcdfcb/CMakeLists.txt#L38-L42

[EsbernTK]

Okay great, then i already have it.
On a sidenote im building on whispercpp and whispercpp.py, to create Cython extension for whisper.cpp with the posibility of enabling the extra features, like CUDA and coreml support, which the other extensions dont have. When im done, can i create a pull request with the link, to add it to the list in the readme?

[bobqianic]

When im done, can i create a pull request with the link, to add it to the list in the readme?

Absolutely! Feel free to create a pull request once you're done, and I'd be happy to add it to the readme list.