Support Qwen3

[ckvv]

Feature Description

https:github.com/ggml-org/llama.cpp/pull/12828 already supports Qwen3

The Solution

update llama version to support Qwen3

Considered Alternatives

update llama version to support Qwen3

Additional Context

No response

Related Features to This Feature Request

• Metal support
• CUDA support
• Vulkan support
• Grammar
• Function calling

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, and I know how to start.

[giladgd]

I'll release a new version with updated llama.cpp that supports Qwen 3 in about a week

[louistiti]

Hi @giladgd, thanks a lot for all the work you've done with node-llama-cpp!
I was wondering if it'd be possible to get a guide or something so in the future we can contribute in supporting such new models?

[ckvv]

Hi @giladgd, thanks a lot for all the work you've done with node-llama-cpp! I was wondering if it'd be possible to get a guide or something so in the future we can contribute in supporting such new models?

Looking at the code, it seems that only need to update ChatWrapper and then rebuild (the latest llama.cpp code will be pulled for building.)

[giladgd]

There are no logic changes needed in node-llama-cpp to support Qwen 3, only to adapt to the breaking changes in llama.cpp to make the latest release compile (which I'm currently working on).
If there weren't any breaking changes in llama.cpp then you could've just built the latest llama.cpp release locally and use a Qwen 3 model right away.
This is true to most models, except ones that need new features in node-llama-cpp to be easier to use, like the chain of thought segmentation I recently added for thinking models.

[louistiti]

Thanks for the reply @giladgd, I actually tried last week, and I may fall in the "If there weren't any breaking changes in llama.cpp" condition 😆

Indeed, on macOS, after running the build command:

npx --no node-llama-cpp source download --release latest

I'm getting the following error log:

Repo: ggml-org/llama.cpp
Release: latest
GPU: Metal

✔ Fetched llama.cpp info
✔ Removed existing llama.cpp directory
✔ Cloned ggml-org/llama.cpp (GitHub)
◷ Compiling llama.cpp
Not searching for unused variables given on the command line.
-- The C compiler identification is AppleClang 17.0.0.17000013
-- The CXX compiler identification is AppleClang 17.0.0.17000013
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.39.5 (Apple Git-154)")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- CMAKE_SYSTEM_PROCESSOR: arm64
-- Including CPU backend
-- Accelerate framework found
-- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
-- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)
CMake Warning at llama.cpp/ggml/src/ggml-cpu/CMakeLists.txt:63 (message):
  OpenMP not found
Call Stack (most recent call first):
  llama.cpp/ggml/src/CMakeLists.txt:302 (ggml_add_cpu_backend_variant_impl)


-- ARM detected
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E - Failed
-- ARM -mcpu not found, -mcpu=native will be used
-- Performing Test GGML_MACHINE_SUPPORTS_dotprod
-- Performing Test GGML_MACHINE_SUPPORTS_dotprod - Success
-- Performing Test GGML_MACHINE_SUPPORTS_i8mm
-- Performing Test GGML_MACHINE_SUPPORTS_i8mm - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_noi8mm
-- Performing Test GGML_MACHINE_SUPPORTS_noi8mm - Success
-- Performing Test GGML_MACHINE_SUPPORTS_sve
-- Performing Test GGML_MACHINE_SUPPORTS_sve - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_nosve
-- Performing Test GGML_MACHINE_SUPPORTS_nosve - Success
-- Performing Test GGML_MACHINE_SUPPORTS_sme
-- Performing Test GGML_MACHINE_SUPPORTS_sme - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_nosme
-- Performing Test GGML_MACHINE_SUPPORTS_nosme - Success
-- ARM feature DOTPROD enabled
-- ARM feature FMA enabled
-- ARM feature FP16_VECTOR_ARITHMETIC enabled
-- Adding CPU backend variant ggml-cpu: -mcpu=native+dotprod+noi8mm+nosve+nosme
-- Looking for dgemm_
-- Looking for dgemm_ - found
-- Found BLAS: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Accelerate.framework
-- BLAS found, Libraries: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Accelerate.framework
-- BLAS found, Includes:
-- Including BLAS backend
-- Metal framework found
-- The ASM compiler identification is AppleClang
-- Found assembler: /usr/bin/cc
-- Including METAL backend
-- Found CURL: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libcurl.tbd (found version "8.7.1")
-- Configuring done (8.7s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/localBuilds/mac-arm64-metal-release-b5342
[1/80] Building C object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml-alloc.c.o
[2/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml-threading.cpp.o
[3/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu-hbm.cpp.o
[4/80] Building C object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml.c.o
[5/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.cpp.o
[6/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml-backend.cpp.o
[7/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml-opt.cpp.o
[8/80] Building C object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu-quants.c.o
[9/80] Building C object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.c.o
[10/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu-traits.cpp.o
[11/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/amx.cpp.o
[12/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/mmq.cpp.o
[13/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/gguf.cpp.o
[14/80] Generating build details from Git
-- Found Git: /usr/bin/git (found version "2.39.5 (Apple Git-154)")
[15/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu-aarch64.cpp.o
[16/80] Building CXX object llama.cpp/common/CMakeFiles/build_info.dir/build-info.cpp.o
[17/80] Building C object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml-quants.c.o
[18/80] Linking CXX shared library bin/libggml-base.dylib
[19/80] Generate assembly for embedded Metal library
Embedding Metal library
[20/80] Building ASM object llama.cpp/ggml/src/ggml-metal/CMakeFiles/ggml-metal.dir/__/__/__/__/autogenerated/ggml-metal-embed.s.o
[21/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/unary-ops.cpp.o
[22/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/vec.cpp.o
[23/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/llamafile/sgemm.cpp.o
[24/80] Building CXX object llama.cpp/ggml/src/ggml-blas/CMakeFiles/ggml-blas.dir/ggml-blas.cpp.o
[25/80] Linking CXX shared library bin/libggml-blas.dylib
[26/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/binary-ops.cpp.o
[27/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama.cpp.o
[28/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml.dir/ggml-backend-reg.cpp.o
[29/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-adapter.cpp.o
[30/80] Building C object llama.cpp/ggml/src/ggml-metal/CMakeFiles/ggml-metal.dir/ggml-metal.m.o
[31/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-batch.cpp.o
[32/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-arch.cpp.o
[33/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-chat.cpp.o
[34/80] Linking C shared library bin/libggml-metal.dylib
[35/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-memory.cpp.o
[36/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-hparams.cpp.o
[37/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ops.cpp.o
[38/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-io.cpp.o
[39/80] Linking CXX shared library bin/libggml-cpu.dylib
[40/80] Linking CXX shared library bin/libggml.dylib
[41/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-graph.cpp.o
[42/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-impl.cpp.o
[43/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-context.cpp.o
[44/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-mmap.cpp.o
[45/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-kv-cache.cpp.o
[46/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-grammar.cpp.o
[47/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-model-loader.cpp.o
[48/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/unicode-data.cpp.o
[49/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-vocab.cpp.o
[50/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-quant.cpp.o
[51/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-sampling.cpp.o
[52/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/console.cpp.o
[53/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/llguidance.cpp.o
[54/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/unicode.cpp.o
[55/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/log.cpp.o
[56/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/common.cpp.o
[57/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-model.cpp.o
[58/80] Linking CXX shared library bin/libllama.dylib
[59/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/ngram-cache.cpp.o
[60/80] Building CXX object CMakeFiles/llama-addon.dir/addon/AddonGrammar.cpp.o
FAILED: CMakeFiles/llama-addon.dir/addon/AddonGrammar.cpp.o
/usr/bin/c++ -DGGML_BACKEND_SHARED -DGGML_SHARED -DGGML_USE_BLAS -DGGML_USE_CPU -DGGML_USE_METAL -DLLAMA_SHARED -DLLAMA_USE_CURL -DNAPI_VERSION=7 -Dllama_addon_EXPORTS -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-addon-api@8.3.1/node_modules/node-addon-api -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-api-headers@1.5.0/node_modules/node-api-headers/include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/./llama.cpp/common -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/ggml/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/common/. -D_DARWIN_USE_64_BIT_INODE=1 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DBUILDING_NODE_EXTENSION -O3 -DNDEBUG -std=gnu++17 -arch arm64 -fPIC -fexceptions -Wno-c++17-extensions -MD -MT CMakeFiles/llama-addon.dir/addon/AddonGrammar.cpp.o -MF CMakeFiles/llama-addon.dir/addon/AddonGrammar.cpp.o.d -o CMakeFiles/llama-addon.dir/addon/AddonGrammar.cpp.o -c '/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammar.cpp'
In file included from /Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammar.cpp:2:
/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammar.h:4:10: fatal error: 'llama-grammar.h' file not found
    4 | #include "llama-grammar.h"
      |          ^~~~~~~~~~~~~~~~~
1 error generated.
[61/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/sampling.cpp.o
[62/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/speculative.cpp.o
[63/80] Building CXX object CMakeFiles/llama-addon.dir/addon/AddonContext.cpp.o
FAILED: CMakeFiles/llama-addon.dir/addon/AddonContext.cpp.o
/usr/bin/c++ -DGGML_BACKEND_SHARED -DGGML_SHARED -DGGML_USE_BLAS -DGGML_USE_CPU -DGGML_USE_METAL -DLLAMA_SHARED -DLLAMA_USE_CURL -DNAPI_VERSION=7 -Dllama_addon_EXPORTS -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-addon-api@8.3.1/node_modules/node-addon-api -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-api-headers@1.5.0/node_modules/node-api-headers/include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/./llama.cpp/common -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/ggml/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/common/. -D_DARWIN_USE_64_BIT_INODE=1 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DBUILDING_NODE_EXTENSION -O3 -DNDEBUG -std=gnu++17 -arch arm64 -fPIC -fexceptions -Wno-c++17-extensions -MD -MT CMakeFiles/llama-addon.dir/addon/AddonContext.cpp.o -MF CMakeFiles/llama-addon.dir/addon/AddonContext.cpp.o.d -o CMakeFiles/llama-addon.dir/addon/AddonContext.cpp.o -c '/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonContext.cpp'
/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonContext.cpp:5:10: fatal error: 'llama-grammar.h' file not found
    5 | #include "llama-grammar.h"
      |          ^~~~~~~~~~~~~~~~~
1 error generated.
[64/80] Building CXX object CMakeFiles/llama-addon.dir/addon/AddonGrammarEvaluationState.cpp.o
FAILED: CMakeFiles/llama-addon.dir/addon/AddonGrammarEvaluationState.cpp.o
/usr/bin/c++ -DGGML_BACKEND_SHARED -DGGML_SHARED -DGGML_USE_BLAS -DGGML_USE_CPU -DGGML_USE_METAL -DLLAMA_SHARED -DLLAMA_USE_CURL -DNAPI_VERSION=7 -Dllama_addon_EXPORTS -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-addon-api@8.3.1/node_modules/node-addon-api -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-api-headers@1.5.0/node_modules/node-api-headers/include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/./llama.cpp/common -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/ggml/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/common/. -D_DARWIN_USE_64_BIT_INODE=1 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DBUILDING_NODE_EXTENSION -O3 -DNDEBUG -std=gnu++17 -arch arm64 -fPIC -fexceptions -Wno-c++17-extensions -MD -MT CMakeFiles/llama-addon.dir/addon/AddonGrammarEvaluationState.cpp.o -MF CMakeFiles/llama-addon.dir/addon/AddonGrammarEvaluationState.cpp.o.d -o CMakeFiles/llama-addon.dir/addon/AddonGrammarEvaluationState.cpp.o -c '/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammarEvaluationState.cpp'
In file included from /Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammarEvaluationState.cpp:6:
/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammar.h:4:10: fatal error: 'llama-grammar.h' file not found
    4 | #include "llama-grammar.h"
      |          ^~~~~~~~~~~~~~~~~
1 error generated.
[65/80] Building CXX object CMakeFiles/llama-addon.dir/addon/AddonModel.cpp.o
[66/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/arg.cpp.o
[67/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/json-schema-to-grammar.cpp.o
[68/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/chat.cpp.o
ninja: build stopped: subcommand failed.
Not searching for unused variables given on the command line.
-- The C compiler identification is AppleClang 17.0.0.17000013
-- The CXX compiler identification is AppleClang 17.0.0.17000013
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.39.5 (Apple Git-154)")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- CMAKE_SYSTEM_PROCESSOR: arm64
-- Including CPU backend
-- Accelerate framework found
-- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
-- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)
CMake Warning at llama.cpp/ggml/src/ggml-cpu/CMakeLists.txt:63 (message):
  OpenMP not found
Call Stack (most recent call first):
  llama.cpp/ggml/src/CMakeLists.txt:302 (ggml_add_cpu_backend_variant_impl)


-- ARM detected
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E - Failed
-- ARM -mcpu not found, -mcpu=native will be used
-- Performing Test GGML_MACHINE_SUPPORTS_dotprod
-- Performing Test GGML_MACHINE_SUPPORTS_dotprod - Success
-- Performing Test GGML_MACHINE_SUPPORTS_i8mm
-- Performing Test GGML_MACHINE_SUPPORTS_i8mm - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_noi8mm
-- Performing Test GGML_MACHINE_SUPPORTS_noi8mm - Success
-- Performing Test GGML_MACHINE_SUPPORTS_sve
-- Performing Test GGML_MACHINE_SUPPORTS_sve - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_nosve
-- Performing Test GGML_MACHINE_SUPPORTS_nosve - Success
-- Performing Test GGML_MACHINE_SUPPORTS_sme
-- Performing Test GGML_MACHINE_SUPPORTS_sme - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_nosme
-- Performing Test GGML_MACHINE_SUPPORTS_nosme - Success
-- ARM feature DOTPROD enabled
-- ARM feature FMA enabled
-- ARM feature FP16_VECTOR_ARITHMETIC enabled
-- Adding CPU backend variant ggml-cpu: -mcpu=native+dotprod+noi8mm+nosve+nosme
-- Looking for dgemm_
-- Looking for dgemm_ - found
-- Found BLAS: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Accelerate.framework
-- BLAS found, Libraries: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Accelerate.framework
-- BLAS found, Includes:
-- Including BLAS backend
-- Metal framework found
-- The ASM compiler identification is AppleClang
-- Found assembler: /usr/bin/cc
-- Including METAL backend
-- Found CURL: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libcurl.tbd (found version "8.7.1")
-- Configuring done (5.3s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/localBuilds/mac-arm64-metal-release-b5342
[1/80] Building C object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml-alloc.c.o
[2/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml-threading.cpp.o
[3/80] Building C object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml.c.o
[4/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu-hbm.cpp.o
[5/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.cpp.o
[6/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml-backend.cpp.o
[7/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml-opt.cpp.o
[8/80] Building C object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu-quants.c.o
[9/80] Building C object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.c.o
[10/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/gguf.cpp.o
[11/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu-traits.cpp.o
[12/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/mmq.cpp.o
[13/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/amx.cpp.o
[14/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu-aarch64.cpp.o
[15/80] Generating build details from Git
-- Found Git: /usr/bin/git (found version "2.39.5 (Apple Git-154)")
[16/80] Building CXX object llama.cpp/common/CMakeFiles/build_info.dir/build-info.cpp.o
[17/80] Building C object llama.cpp/ggml/src/CMakeFiles/ggml-base.dir/ggml-quants.c.o
[18/80] Linking CXX shared library bin/libggml-base.dylib
[19/80] Generate assembly for embedded Metal library
Embedding Metal library
[20/80] Building ASM object llama.cpp/ggml/src/ggml-metal/CMakeFiles/ggml-metal.dir/__/__/__/__/autogenerated/ggml-metal-embed.s.o
[21/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/unary-ops.cpp.o
[22/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/vec.cpp.o
[23/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/llamafile/sgemm.cpp.o
[24/80] Building CXX object llama.cpp/ggml/src/ggml-blas/CMakeFiles/ggml-blas.dir/ggml-blas.cpp.o
[25/80] Linking CXX shared library bin/libggml-blas.dylib
[26/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama.cpp.o
[27/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml.dir/ggml-backend-reg.cpp.o
[28/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/binary-ops.cpp.o
[29/80] Building C object llama.cpp/ggml/src/ggml-metal/CMakeFiles/ggml-metal.dir/ggml-metal.m.o
[30/80] Linking C shared library bin/libggml-metal.dylib
[31/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-adapter.cpp.o
[32/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-batch.cpp.o
[33/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-chat.cpp.o
[34/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-arch.cpp.o
[35/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-memory.cpp.o
[36/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-hparams.cpp.o
[37/80] Building CXX object llama.cpp/ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ops.cpp.o
[38/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-graph.cpp.o
[39/80] Linking CXX shared library bin/libggml-cpu.dylib
[40/80] Linking CXX shared library bin/libggml.dylib
[41/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-io.cpp.o
[42/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-context.cpp.o
[43/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-impl.cpp.o
[44/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-mmap.cpp.o
[45/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-grammar.cpp.o
[46/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-kv-cache.cpp.o
[47/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-model-loader.cpp.o
[48/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/unicode-data.cpp.o
[49/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-vocab.cpp.o
[50/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-quant.cpp.o
[51/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-sampling.cpp.o
[52/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/console.cpp.o
[53/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/llguidance.cpp.o
[54/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/unicode.cpp.o
[55/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/log.cpp.o
[56/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/common.cpp.o
[57/80] Building CXX object llama.cpp/src/CMakeFiles/llama.dir/llama-model.cpp.o
[58/80] Linking CXX shared library bin/libllama.dylib
[59/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/speculative.cpp.o
[60/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/ngram-cache.cpp.o
[61/80] Building CXX object CMakeFiles/llama-addon.dir/addon/AddonContext.cpp.o
FAILED: CMakeFiles/llama-addon.dir/addon/AddonContext.cpp.o
/usr/bin/c++ -DGGML_BACKEND_SHARED -DGGML_SHARED -DGGML_USE_BLAS -DGGML_USE_CPU -DGGML_USE_METAL -DLLAMA_SHARED -DLLAMA_USE_CURL -DNAPI_VERSION=7 -Dllama_addon_EXPORTS -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-addon-api@8.3.1/node_modules/node-addon-api -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-api-headers@1.5.0/node_modules/node-api-headers/include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/./llama.cpp/common -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/ggml/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/common/. -D_DARWIN_USE_64_BIT_INODE=1 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DBUILDING_NODE_EXTENSION -O3 -DNDEBUG -std=gnu++17 -arch arm64 -fPIC -fexceptions -Wno-c++17-extensions -MD -MT CMakeFiles/llama-addon.dir/addon/AddonContext.cpp.o -MF CMakeFiles/llama-addon.dir/addon/AddonContext.cpp.o.d -o CMakeFiles/llama-addon.dir/addon/AddonContext.cpp.o -c '/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonContext.cpp'
/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonContext.cpp:5:10: fatal error: 'llama-grammar.h' file not found
    5 | #include "llama-grammar.h"
      |          ^~~~~~~~~~~~~~~~~
1 error generated.
[62/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/sampling.cpp.o
[63/80] Building CXX object CMakeFiles/llama-addon.dir/addon/AddonGrammar.cpp.o
FAILED: CMakeFiles/llama-addon.dir/addon/AddonGrammar.cpp.o
/usr/bin/c++ -DGGML_BACKEND_SHARED -DGGML_SHARED -DGGML_USE_BLAS -DGGML_USE_CPU -DGGML_USE_METAL -DLLAMA_SHARED -DLLAMA_USE_CURL -DNAPI_VERSION=7 -Dllama_addon_EXPORTS -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-addon-api@8.3.1/node_modules/node-addon-api -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-api-headers@1.5.0/node_modules/node-api-headers/include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/./llama.cpp/common -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/ggml/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/common/. -D_DARWIN_USE_64_BIT_INODE=1 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DBUILDING_NODE_EXTENSION -O3 -DNDEBUG -std=gnu++17 -arch arm64 -fPIC -fexceptions -Wno-c++17-extensions -MD -MT CMakeFiles/llama-addon.dir/addon/AddonGrammar.cpp.o -MF CMakeFiles/llama-addon.dir/addon/AddonGrammar.cpp.o.d -o CMakeFiles/llama-addon.dir/addon/AddonGrammar.cpp.o -c '/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammar.cpp'
In file included from /Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammar.cpp:2:
/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammar.h:4:10: fatal error: 'llama-grammar.h' file not found
    4 | #include "llama-grammar.h"
      |          ^~~~~~~~~~~~~~~~~
1 error generated.
[64/80] Building CXX object CMakeFiles/llama-addon.dir/addon/AddonGrammarEvaluationState.cpp.o
FAILED: CMakeFiles/llama-addon.dir/addon/AddonGrammarEvaluationState.cpp.o
/usr/bin/c++ -DGGML_BACKEND_SHARED -DGGML_SHARED -DGGML_USE_BLAS -DGGML_USE_CPU -DGGML_USE_METAL -DLLAMA_SHARED -DLLAMA_USE_CURL -DNAPI_VERSION=7 -Dllama_addon_EXPORTS -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-addon-api@8.3.1/node_modules/node-addon-api -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-api-headers@1.5.0/node_modules/node-api-headers/include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/./llama.cpp/common -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/ggml/src/../include -I/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/llama.cpp/common/. -D_DARWIN_USE_64_BIT_INODE=1 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DBUILDING_NODE_EXTENSION -O3 -DNDEBUG -std=gnu++17 -arch arm64 -fPIC -fexceptions -Wno-c++17-extensions -MD -MT CMakeFiles/llama-addon.dir/addon/AddonGrammarEvaluationState.cpp.o -MF CMakeFiles/llama-addon.dir/addon/AddonGrammarEvaluationState.cpp.o.d -o CMakeFiles/llama-addon.dir/addon/AddonGrammarEvaluationState.cpp.o -c '/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammarEvaluationState.cpp'
In file included from /Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammarEvaluationState.cpp:6:
/Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/llama/addon/AddonGrammar.h:4:10: fatal error: 'llama-grammar.h' file not found
    4 | #include "llama-grammar.h"
      |          ^~~~~~~~~~~~~~~~~
1 error generated.
[65/80] Building CXX object CMakeFiles/llama-addon.dir/addon/AddonModelData.cpp.o
[66/80] Building CXX object CMakeFiles/llama-addon.dir/addon/AddonModel.cpp.o
[67/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/json-schema-to-grammar.cpp.o
[68/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/arg.cpp.o
[69/80] Building CXX object llama.cpp/common/CMakeFiles/common.dir/chat.cpp.o
ninja: build stopped: subcommand failed.
ERR! OMG Process terminated: 1
✖️ Failed to compile llama.cpp
Failed to build llama.cpp with Metal support. Error: SpawnError: Command npm run -s cmake-js-llama -- compile --log-level warn --config Release --arch=arm64 --out localBuilds/mac-arm64-metal-release-b5342 --runtime-version=22.14.0 --parallel=9 --CDGGML_BUILD_NUMBER=1 --CDCMAKE_CONFIGURATION_TYPES=Release --CDNLC_CURRENT_PLATFORM=mac-arm64 --CDNLC_TARGET_PLATFORM=mac-arm64 --CDGGML_METAL=1 --CDGGML_CCACHE=OFF exited with code 1
    at createError (file:///Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/dist/utils/spawnCommand.js:34:20)
    at ChildProcess.<anonymous> (file:///Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/dist/utils/spawnCommand.js:47:24)
    at ChildProcess.emit (node:events:518:28)
    at ChildProcess._handle.onexit (node:internal/child_process:293:12)
node-llama-cpp source download

Download a release of `llama.cpp` and compile it
https://node-llama-cpp.withcat.ai/cli/source/download

Options:
  -h, --help                  Show help                                                                          [boolean]
      --repo                  The GitHub repository to download a release of llama.cpp from. Can also be set via the `NODE
                              _LLAMA_CPP_REPO` environment variable               [string] [default: "ggml-org/llama.cpp"]
      --release               The tag of the llama.cpp release to download. Set to `latest` to download the latest release
                              . Can also be set via the `NODE_LLAMA_CPP_REPO_RELEASE` environment variable
                                                                                               [string] [default: "b4980"]
  -a, --arch                  The architecture to compile llama.cpp for                                           [string]
  -t, --nodeTarget            The Node.js version to compile llama.cpp for. Example: `v18.0.0`                    [string]
      --gpu                   Compute layer implementation type to use for llama.cpp
                                            [string] [choices: "auto", "metal", "cuda", "vulkan", false] [default: "auto"]
      --skipBuild, --sb       Skip building llama.cpp after downloading it                      [boolean] [default: false]
      --noBundle, --nb        Download a llama.cpp release only from GitHub, even if a local git bundle exists for the rel
                              ease                                                              [boolean] [default: false]
      --noUsageExample, --nu  Don't print code usage example after building                     [boolean] [default: false]
  -v, --version               Show version number                                                                [boolean]

SpawnError: Command npm run -s cmake-js-llama -- compile --log-level warn --config Release --arch=arm64 --out localBuilds/mac-arm64-metal-release-b5342 --runtime-version=22.14.0 --parallel=9 --CDGGML_BUILD_NUMBER=1 --CDCMAKE_CONFIGURATION_TYPES=Release --CDNLC_CURRENT_PLATFORM=mac-arm64 --CDNLC_TARGET_PLATFORM=mac-arm64 --CDGGML_METAL=1 --CDGGML_CCACHE=OFF exited with code 1
    at createError (file:///Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/dist/utils/spawnCommand.js:34:20)
    at ChildProcess.<anonymous> (file:///Users/louistiti/Workspace/myproject/node_modules/.pnpm/node-llama-cpp@3.7.0/node_modules/node-llama-cpp/dist/utils/spawnCommand.js:47:24)
    at ChildProcess.emit (node:events:518:28)
    at ChildProcess._handle.onexit (node:internal/child_process:293:12)

It seems that the llama-grammar.h header file cannot be found.

[github-actions]

🎉 This issue has been resolved in version 3.8.0 🎉

The release is available on:

• v3.8.0
• npm package (@latest dist-tag)
• GitHub release
• v3.8.0

Your semantic-release bot 📦🚀