"ggml-metal.metal" file couldn't be found when loading the large-v3 model for CoreML

[Gdesau]

Hello everyone,
I'm working with an M3 Max and I've tried to load the "ggml-large-v3.bin" model with the following code:

from pywhispercpp.model import Model
model = Model('/Users/my_user/Dev/Models/Whisper_large_v3/ggml-large-v3.bin', n_threads=6)
print(Model.system_info())  # and you should see COREML = 1

But it's unable to find the ggml-metal.metal file when it is actually present in the whisper.cpp folder. It gives me the following result:

[2024-04-30 17:38:52,675] {model.py:221} INFO - Initializing the model ...
AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | METAL = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | CUDA = 0 | COREML = 1 | OPENVINO = 0

whisper_init_from_file_with_params_no_state: loading model from '/Users/my_user/Dev/Models/Whisper_large_v3/ggml-large-v3.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Max
ggml_metal_init: picking default device: Apple M3 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: error: could not use bundle path to find ggml-metal.metal, falling back to trying cwd
ggml_metal_init: loading 'ggml-metal.metal'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=260 "The file “ggml-metal.metal” couldn’t be opened because there is no such file." UserInfo={NSFilePath=ggml-metal.metal, NSUnderlyingError=0x60000250f690 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
whisper_backend_init: ggml_backend_metal_init() failed
whisper_model_load:      CPU total size =  3094.36 MB
whisper_model_load: model size    = 3094.36 MB
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Max
ggml_metal_init: picking default device: Apple M3 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: error: could not use bundle path to find ggml-metal.metal, falling back to trying cwd
ggml_metal_init: loading 'ggml-metal.metal'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=260 "The file “ggml-metal.metal” couldn’t be opened because there is no such file." UserInfo={NSFilePath=ggml-metal.metal, NSUnderlyingError=0x600002508f00 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
whisper_backend_init: ggml_backend_metal_init() failed
whisper_init_state: kv self size  =  220.20 MB
whisper_init_state: kv cross size =  245.76 MB
whisper_init_state: loading Core ML model from '/Users/my_user/Dev/Models/Whisper_large_v3/ggml-large-v3-encoder.mlmodelc'
whisper_init_state: first run on a device may take a while ...
whisper_init_state: Core ML model loaded
whisper_init_state: compute buffer (conv)   =   10.92 MB
whisper_init_state: compute buffer (cross)  =    9.38 MB
whisper_init_state: compute buffer (decode) =  209.26 MB

I've tried to add the path to the environment variables with:

export GGML_METAL_PATH_RESOURCES=/Users/gregoiredesauvage/Dev/Modules/pywhispercpp/whisper.cpp/ggml-metal.metal

but it didn't work.

I have the "ggml-large-v3-encoder.mlmodelc" file in the same folder as the "ggml-large-v3.bin" file.

Any idea?

[abdeladim-s]

Hello @Gdesau,

According to this I think you need to export the folder where the ggml-metal.metal file is located, not the file itself,
So in your case, try something like:

export GGML_METAL_PATH_RESOURCES=/Users/gregoiredesauvage/Dev/Modules/pywhispercpp/whisper.cpp

If it didn't work, maybe you can try the following as well:

GGML_METAL_PATH_RESOURCES=/Users/gregoiredesauvage/Dev/Modules/pywhispercpp/whisper.cpp python /path/to/your/script.py

[Gdesau]

Hello @abdeladim-s,

Thank you very much for your help. Unfortunately the first solution couldn't fix my issue and I couldn't try the second one because I'm working in a notebook. But I'm thinking that maybe the issue may come from these 2 lines in the result:

ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil

These are located just after this line:

ggml_metal_init: picking default device: Apple M3 Max

I've tried to search about the default.metallib and why it's not found but couldn't find something helpful.

Do you have any idea?

[abdeladim-s]

@Gdesau, I think using a notebook is part of the problem.
GGML_METAL_PATH_RESOURCES won't visible inside the notebook with the export command, as you can see from the log

ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil

What I can suggest:

• Preferably, try running the script in a command line.
• Try to set the env variable inside the notebook, using %env GGML_METAL_PATH_RESOURCES=/Users/gregoiredesauvage/Dev/Modules/pywhispercpp/whisper.cpp, or modify the kernel.json file and see if that will work.

[Gdesau]

@abdeladim-s, thanks a lot, setting the env variable with %env GGML_METAL_PATH_RESOURCES=/Users/gregoiredesauvage/Dev/Modules/pywhispercpp/whisper.cpp inside the notebook solved the problem! But I ran into another problem then, the "ggml-common.h" file has not been found but it's also located in the /Users/my_user/Dev/Modules/pywhispercpp/whisper.cpp directory:

ggml_metal_init: error: Error Domain=MTLLibraryErrorDomain Code=3 "program_source:3:10: fatal error: 'ggml-common.h' file not found
#include "ggml-common.h"
         ^~~~~~~~~~~~~~~
" UserInfo={NSLocalizedDescription=program_source:3:10: fatal error: 'ggml-common.h' file not found
#include "ggml-common.h"
         ^~~~~~~~~~~~~~~
}
whisper_backend_init: ggml_backend_metal_init() failed

Do you think that there are other env variables to set? I've checked the "ggml-metal.metal" file it didn't seem to be but maybe I missed something

[abdeladim-s]

@Gdesau, I think it's the same issue as #2041
Are you using the latest commit from whisper.cpp ?
Please follow the instructions on the readme page and build against the same hash commit included in the project.

[Gdesau]

@abdeladim-s, Thanks a lot! I followed the #2041, and hardcoded the path to the "ggml-common.h" file into the "ggml-metal.metal" file and it worked! So the 3 first lines of "ggml-metal.metal" file were as follows:

#define GGML_COMMON_DECL_METAL
#define GGML_COMMON_IMPL_METAL
#include "ggml-common.h"

And I changed 3rd line with:

#include "/Users/my_user/Dev/Modules/pywhispercpp/whisper.cpp/ggml-common.h"

Now the Core ML model is loaded successfully in my notebook. So, to summarize, the full solution is to:

1. Replace the 3rd line of the "ggml-metal.metal" as explained above
2. Set the GGML_METAL_PATH_RESOURCES env variable in the notebook with the path to the directory containing the "ggml-metal.metal" file:

%env GGML_METAL_PATH_RESOURCES=/Users/my_user/Dev/Modules/pywhispercpp/whisper.cpp

[abdeladim-s]

`Glad it finally worked!
Thanks @Gdesau for posting the solution, this will certainly help others facing the same issue.