Error running on M1 Mac

[DrBenjamin]

Hi!

I am having issues with using it on a M1 Mac:

from llama_cpp import Llama
produces this error:

zsh: illegal hardware instruction

Best,
Benjamin

[abetlen]

Do you mind posting the complete error message? I'm assuming this occured when importing the package. I'm currently using cmake to specify the llama.cpp project as a t

Can you try installing the package from source? Just run the following commands in a terminal

git clone https://github.com/abetlen/llama-cpp-python.git
cd llama-cpp-python
python3 setup.py develop

EDIT: Additionally (maybe good to check first), please confirm that llama.cpp can compile and run by running:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make
./main -m ./models/7B/ggml-model-q4_0.bin -n 128

[DrBenjamin]

The install from source with

python3 setup.py develop

runs into this error:

CMake Error at CMakeLists.txt:7 (add_subdirectory):
  The source directory

    /Users/ben/Downloads/llama-cpp-python/vendor/llama.cpp

  does not contain a CMakeLists.txt file.


CMake Error at CMakeLists.txt:9 (install):
  install TARGETS given target "llama" which does not exist.


-- Configuring incomplete, errors occurred!
Traceback (most recent call last):
  File "/Users/ben/opt/miniconda3.10/miniconda3/lib/python3.10/site-packages/skbuild/setuptools_wrap.py", line 634, in setup
    env = cmkr.configure(
  File "/Users/ben/opt/miniconda3.10/miniconda3/lib/python3.10/site-packages/skbuild/cmaker.py", line 332, in configure
    raise SKBuildError(

llama.cpp did go through the compilation, but

./main -m models/models/7B/ggml-model-q4_0.bin -n 128

throws this error:

main: seed = 1679966146
llama_model_load: loading model from 'models/models/7B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: invalid model file 'models/models/7B/ggml-model-q4_0.bin' (too old, regenerate your model files!)
llama_init_from_file: failed to load model
main: error: failed to load model 'models/models/7B/ggml-model-q4_0.bin'

That is weird as dalai (which downloaded the alpaca 7B model) runs without any problems with this model. Also this script works fine https://github.com/shaunabanana/llama.py with the model.

[DrBenjamin]

Ok, I learned that I need to put the llama.cpp files into .vendor/llama.cpp. Then it worked. Giving me this error:

OSError: dlopen(/Users/ben/opt/miniconda3.10/miniconda3/lib/python3.10/site-packages/llama_cpp/libllama.dylib, 0x0006): tried: '/Users/ben/opt/miniconda3.10/miniconda3/lib/python3.10/site-packages/llama_cpp/libllama.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')),

'/System/Volumes/Preboot/Cryptexes/OS/Users/ben/opt/miniconda3.10/miniconda3/lib/python3.10/site-packages/llama_cpp/libllama.dylib' (no such file), '/Users/ben/opt/miniconda3.10/miniconda3/lib/python3.10/site-packages/llama_cpp/libllama.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

[abetlen]

CMake Error at CMakeLists.txt:7 (add_subdirectory):
  The source directory

    /Users/ben/Downloads/llama-cpp-python/vendor/llama.cpp

  does not contain a CMakeLists.txt file.

My bad, I forgot that vendor/llama.cpp is a git submodule so it doesn't get cloned automatically (shouldn't be an issue with the PyPI version though). Running git submodule init and git submodule update after cloning the repo should work.

main: seed = 1679966146
llama_model_load: loading model from 'models/models/7B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: invalid model file 'models/models/7B/ggml-model-q4_0.bin' (too old, regenerate your model files!)
llama_init_from_file: failed to load model
main: error: failed to load model 'models/models/7B/ggml-model-q4_0.bin'

The error you're getting here is because llama.cpp recently updated the binary format for the model weights. I believe dalai is using an older version of llama.cpp which is why it's not giving you an error. You'll need to run the convert model and quantize steps on the original weights documented here.

python3 convert-pth-to-ggml.py models/7B/ 1
python3 quantize.py 7B

Finally, back to our original error

(mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')),

I've actually tested the install (through pip not conda) on an M1 Macbook Air and it all worked, this seems to be an issue with the Mac M1s x86_64 compatibility mode. I think it has to do with installing it through conda but I'm not sure. I'll try conda installing the repo tomorrow and see what happens.

In the meantime can you try running arch -x86_64 before running the script?

[DrBenjamin]

arch -x86_64 leads to the same error as in the beginning:

arch -x86_64 python Test.py  
zsh: illegal hardware instruction  arch -x86_64 python Test.py

[DrBenjamin]

After converting the model and quantizing steps on the original weights

./main -m models/models/7B/ggml-model-q4_0.bin -n 128

is running. Thanks for this so far!

[pjlegato]

The "illegal instruction" errors usually mean that the binary was compiled with an extension instruction set specified which is not supported by the current CPU at runtime. Look for a line in the makefile with flags similar to -msse3. These -m instructions toggle on various instruction set extensions (see https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html for details.)

Unless there's a specific reason to choose the extensions manually (like cross-compiling for some other CPU), it's usually better to specify -march=native rather than manually specifying instruction set extensions. -march=native auto-selects the best extensions available on the current CPU.

tl;dr remove all those -mwhatevers in the Makefile. Add -march=native where they were. Clean and rebuild, and it should work.

[pjlegato]

@abetlen Expanding on the above, if a user installs a precompiled llama.cpp via pip install llama-cpp-python, they get a binary that often has been compiled on a different CPU than the one they are running on. Since llama.cpp specifies -march=native, it selects instruction set extensions for the compile-time CPU. If the end user has a different CPU with different extensions available, they will get the "illegal instruction" errors above.

The options are:

• Disable extensions altogether, get a slower llama.cpp, but which runs everywhere;
• Recompile llama.cpp upon each installation (somehow) for that user's specific CPU, using whatever extensions are supported there;

[abetlen]

@pjlegato thank you for the info!

wrt the PyPI package I'm actually not distributing binaries for this reason, each shared library should be built on installation which is why this issue is really confusing me.

Currently the package is using scikit-build and building llama.cpp as a shared library via cmake. Some people have pointed out that using make may be better on some systems but I wasn't aware of any bugs using one vs. the other, just assumed some targetted certain optimizations better.

[pjlegato]

I'm not sure, either. If I rebuild llama-cpp-python from source, I get a working llama.cpp main binary in vendor/llama.cpp.

If I use text-generation-webui's --cpu flag, which delegates to an auto-installed llama-cpp-python==0.1.23 in its requirements.txt, I get "illegal instruction" errors.

Maybe some package distributor upstream is somehow pre-buiilding the package and not rebuilding it locally each time?

[abetlen]

I think I found the answer (or at least the root of the issue) here https://github.com/saharNooby/rwkv.cpp#linux--macos

[pjlegato]

Wild guess: I've always compiled llama.cpp on Mac and Linux with plain make (no C), which seems to work.

llama-cpp-python probably compiles llama.cpp by using CMake rather than plain make.

It appears that upstream llama.cpp's CMakeLists.txt tries to be clever and select instruction set extensions manually, rather than just using -march=native: https://github.com/ggerganov/llama.cpp/blob/master/CMakeLists.txt#L193-L219

Could this be it?

[pjlegato]

I am not familiar with CMake. If there's some way you can override llama.cpp's CMakeLists.txt at build time so that LLAMA_NATIVE is ON and all the instruction set selectors at line 54 are OFF, that should fix it.

[abetlen]

Hmm, I may then just move to using make for Mac and Linux (skbuild should support this) and have Windows install via cmake.

@pjlegato just want to confirm, when you've compiled llama.cpp with just make, you haven't had to set any special environment variables?