Llamacpp compile failed on Jetson Orin Nano (8GB)

[wsmzsl]

My nano (8g) is flashed with jetpack6.0 (cuda=12.2, gcc=11.4). When I compile the source code of llama.cpp (with cuda) on orin nano, the following error occurs. Has anyone successfully compiled on nano? Does anyone know how to solve this error?

/usr/lib/gcc/aarch64-linux-gnu/11/include/arm_neon.h(38): error: identifier "__Int8x8_t" is undefined
typedef __Int8x8_t int8x8_t;
^
/usr/lib/gcc/aarch64-linux-gnu/11/include/arm_neon.h(39): error: identifier "__Int16x4_t" is undefined
typedef __Int16x4_t int16x4_t;
^
/usr/lib/gcc/aarch64-linux-gnu/11/include/arm_neon.h(40): error: identifier "__Int32x2_t" is undefined
typedef __Int32x2_t int32x2_t;
^
/usr/lib/gcc/aarch64-linux-gnu/11/include/arm_neon.h(41): error: identifier "__Int64x1_t" is undefined
typedef __Int64x1_t int64x1_t;
^
/usr/lib/gcc/aarch64-linux-gnu/11/include/arm_neon.h(42): error: identifier "__Float16x4_t" is undefined
typedef __Float16x4_t float16x4_t;
^
/usr/lib/gcc/aarch64-linux-gnu/11/include/arm_neon.h(43): error: identifier "__Float32x2_t" is undefined
typedef __Float32x2_t float32x2_t;
^
/usr/lib/gcc/aarch64-linux-gnu/11/include/arm_neon.h(44): error: identifier "__Poly8x8_t" is undefined
typedef __Poly8x8_t poly8x8_t;
^
/usr/lib/gcc/aarch64-linux-gnu/11/include/arm_neon.h(45): error: identifier "__Poly16x4_t" is undefined
typedef __Poly16x4_t poly16x4_t;
......
Error limit reached.
100 errors detected in the compilation of "/home/nano8g-1/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu".
Compilation terminated.
gmake[2]: *** [ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/build.make:314: ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/ggml-cuda.cu.o] Error 4
gmake[1]: *** [CMakeFiles/Makefile2:1760: ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2

[Romyull-Islam]

I am facing a similar problem when building llama.cpp for CPU in the jetson or nano. Lllama.cpp can not be built for Cortex a-78AE CPU.
I think a similar type of issue is occurring here.

[joseph777111]

I don't have this platform; however, it would be much more helpful if you guys would post the exact commands you used to try to compile and run llama.cpp on your respective platform(s). 🤔

Did you follow the instructions from https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md#cuda

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

[Romyull-Islam]

Yeah,
I tried the same commands. But failed.
Then, I had to update the CmakeList.txt file from "ggml/src/ggml-cpu/CMakeLists.txt" for the Cortex A78AE. A78AE was not working. Then selected A76AE. It worked.

Replace the entire if (GGML_NATIVE) execute_process() block with:

if (GGML_NATIVE) # Directly set the CPU architecture to Cortex-A76AE set(ARM_MCPU_FLAG "-mcpu=cortex-a76ae") message(STATUS "ARM -mcpu=cortex-a76ae will be used for optimization") else() # Fallback to generic architecture if needed set(ARM_MCPU_FLAG "-mcpu=generic") message(WARNING "ARM -mcpu=generic will be used as a fallback") endif()

Then, in the newly created build directory, I used the following command for Cuda
cmake .. -DGGML_CUDA=ON -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc -DGGML_CUDA_ARCHS="80" make -j$(nproc)

For CPU only, we can use
cmake .. -DGGML_CUDA=OFF make -j$(nproc)

[joseph777111]

Nice! So you got it to run then? 🤔

[Romyull-Islam]

Nice! So you got it to run then? 🤔

Yeah. It is working. The Cuda is not performing well; seems stuck. so using the CPU right now.
CPU is pretty fast for small model inference.

[joseph777111]

Nice! 😋

[hitmachineWu]

Nice! So you got it to run then? 🤔

Yeah. It is working. The Cuda is not performing well; seems stuck. so using the CPU right now. CPU is pretty fast for small model inference.

Congratulations, but may I ask what versions of gcc and g++ you have locally compiled? I also want to compile the entire project on the Jetson Xavier board, but I encountered some compilation errors. Have you ever come across this issue on your side?

In file included from /home/agilex/junyuan/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-quants.c:7:0:
/home/agilex/junyuan/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-quants.c: In function ‘ggml_vec_dot_q2_K_q8_K’:
/home/agilex/junyuan/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-impl.h:290:27: error: implicit declaration of function ‘vld1q_s16_x2’; did you mean ‘vld1q_s16’? [-Werror=implicit-function-declaration]
 #define ggml_vld1q_s16_x2 vld1q_s16_x2
                           ^

                                         ^~~~~~~~~~~~~~~~~
/home/agilex/junyuan/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-impl.h:290:27: error: invalid initializer
 #define ggml_vld1q_s16_x2 vld1q_s16_x2
                           ^
/home/agilex/junyuan/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-quants.c:4217:41: note: in expansion of macro ‘ggml_vld1q_s16_x2’
         const ggml_int16x8x2_t q8sums = ggml_vld1q_s16_x2(y[i].bsums);
                                         ^~~~~~~~~~~~~~~~~
/home/agilex/junyuan/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-impl.h:291:27: error: implicit declaration of function ‘vld1q_u8_x2’; did you mean ‘vld1q_u32’? [-Werror=implicit-function-declaration]
 #define ggml_vld1q_u8_x2  vld1q_u8_x2
                           ^
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/agilex/junyuan/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-quants.c:7:0:
/home/agilex/junyuan/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-quants.c: In function ‘ggml_vec_dot_q3_K_q8_K’:
/home/agilex/junyuan/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-impl.h:291:27: error: invalid initializer
 #define ggml_vld1q_u8_x2  vld1q_u8_x2
                           ^
/home/agilex/junyuan/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-quants.c:4801:36: note: in expansion of macro ‘ggml_vld1q_u8_x2’
         ggml_uint8x16x2_t qhbits = ggml_vld1q_u8_x2(qh);
                                    ^~~~~~~~~~~~~~~~
/home/agilex/junyuan/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-impl.h:291:27: error: invalid initializer
 #define ggml_vld1q_u8_x2  vld1q_u8_x2