feat: Qualcomm Hexagon NPU support (Android + Windows ARM + Linux ARM) #5

Dependencia

Bloqueante: Ericson246/llama.cpp#2 ÔÇö builds de llama.cpp con -DGGML_HEXAGON=ON

Hasta que existan los builds en el fork, npu-optimize no puede a├▒adir estos runtimes.

Scope de npu-optimize

Una vez que el fork publique builds hexagon, agregar:

1. Hardware detection

Detectar NPU Qualcomm Hexagon en:

Android:

• stat /dev/adsprpc-sdsp (FastRPC device)
• dlopen("libcdsprpc.so") (ADSP daemon library)

Windows ARM64:

• Detectar NPU driver via SetupDiGetClassDevs
• LoadLibrary("libcdsprpc.dll")

Linux ARM64:

• stat /sys/class/remoteproc/remoteproc*/firmware (cdsp remoteproc)
• LoadLibrary("libcdsprpc.so")

2. Catalog entries

Tres nuevos runtimes en Ericson246/llama.cpp:

• android-hexagon-arm64
• windows-hexagon-arm64
• linux-hexagon-arm64

3. Priority update

Selection engine: Hexagon NPU > Vulkan > CPU en Android y Windows ARM64.

4. Schema update

A├▒adir backend "hexagon" a runtime_recommendation.

Summary Add support for Qualcomm Hexagon NPU on Snapdragon devices. llama.cpp upstream already has an experimental ggml-hexagon backend (PR #16547, merged Oct 2025) that uses direct FastRPC to the Hexagon Tensor Processor (HTP). No QNN SDK needed. ## Hardware ### Smartphones (Android) | SoC | Hexagon Arch | Devices | |-----|:-----------:|---------| | Snapdragon 8 Gen 2 | v73 | Galaxy S23, OnePlus 11, Xiaomi 13 | | Snapdragon 8 Gen 3 | v73 | Galaxy S24, OnePlus 12 | | Snapdragon 8s Gen 3 | v75 | Mid-tier 2025+ devices | | Snapdragon 8 Elite (Gen 4) | v79 | Galaxy S25, OnePlus 13 | | Snapdragon 8 Elite Gen 5 | v81 | Galaxy S26+ | ### Laptops (Windows ARM) | SoC | NPU TOPS | Laptops | |-----|:--------:|---------| | Snapdragon X Plus (X1P) | 45 | Surface Pro, ASUS ProArt PZ13 | | Snapdragon X Elite (X1E) | 45 | Surface Laptop 7, ThinkPad T14s Gen 6, XPS 13 9345 | | Snapdragon X2 Elite | ~80 | 2026+ laptops | ### Linux ARM - Snapdragon X Elite (X1E78100) ├ö├ç├ ThinkPad T14s Gen 6, dev kits - Requires mainline kernel 6.12+ with cdsp remoteproc firmware ## What upstream llama.cpp provides - ggml-hexagon backend (-DGGML_HEXAGON=ON): Direct FastRPC to NPU, merged upstream, actively developed by Qualcomm - Multi-platform: Android + Windows ARM64 + Linux ARM64 - Supported ops: MUL_MAT, ADD, RMS_NORM, ROPE, GLU, SOFTMAX, Flash Attention (gen only) - Quantization: Q4_0, Q8_0, MXFP4, FP32 (NOT Q4_K_M, Q5_, Q6_, IQ*) - Perf (Snapdragon X Elite, Llama 3.1 8B Q4_0): ~24 tok/s gen, 720 tok/s prefill ## What upstream does NOT provide - No pre-built binaries with ggml-hexagon enabled ├ö├ç├ only CPU and Vulkan builds are published - No QNN backend merged ├ö├ç├ QNN SDK approach (PR #12063) is still a draft, not in upstream - No Hexagon NPU detection in llama.cpp ├ö├ç├ user must manually specify --device HTP0 ## What npu-optimize needs ### 1. Hardware detection Detect Hexagon NPU availability on each platform: Android: - Check for /dev/adsprpc-sdsp character device (Qualcomm ADSP FastRPC) - Check for libcdsprpc.so in system libraries - Check ro.soc.model = "Snapdragon" or parse /proc/cpuinfo for Qualcomm CPU implementer - Map SoC to Hexagon architecture version (v73/v75/v79/v81) Windows ARM64: - Check for NPU driver: Get-CimInstance Win32_PnPEntity | Where-Object {$_.PNPClass -eq "System" -and $_.Name -match "Qualcomm.*NPU|HTP"} - Check for libQnnHtp.dll or libcdsprpc.so in system32 - Check registry for Qualcomm NPU presence Linux ARM64: - Check for cdsp remoteproc: ls /sys/class/remoteproc/remoteproc*/firmware - Check for FastRPC device: ls /dev/adsprpc-sdsp - Check for libcdsprpc.so in library path - Check CPU implementer for "0x51" (Qualcomm) ### 2. Runtime builds needed Three new runtimes to produce in Ericson246/llama.cpp: #### android-arm64-hexagon - Build: Docker toolchain ghcr.io/snapdragon-toolchain/arm64-android:v0.7 - Flags: -DGGML_HEXAGON=ON -DGGML_OPENCL=ON -DGGML_OPENMP=OFF -DGGML_LLAMAFILE=OFF - Dependencies: libggml-hexagon.so, libggml-htp-v{73,75,79,81}.so, libggml-opencl.so - Device push: adb push pkg-snapdragon /data/local/tmp/ - Runtime: LD_LIBRARY_PATH=./lib ADSP_LIBRARY_PATH=./lib ./bin/llama-server --no-mmap --device HTP0 #### windows-arm64-hexagon - Build: Native ARM64 Windows with MSVC 2026 + Hexagon SDK 6.6+ - Flags: -DGGML_HEXAGON=ON -DGGML_OPENCL=ON -DGGML_OPENMP=OFF -DGGML_LLAMAFILE=OFF - Extra: Test-signing mode required, self-signed cert needed - Runtime: $env:ADSP_LIBRARY_PATH=".\lib" .\bin\llama-server.exe --device HTP0 #### linux-arm64-hexagon - Build: Docker toolchain ghcr.io/snapdragon-toolchain/arm64-linux:v0.1 (cross-compile from x86_64) - Flags: -DGGML_HEXAGON=ON -DGGML_OPENCL=OFF -DGGML_OPENMP=OFF -DGGML_LLAMAFILE=OFF - Runtime: LD_LIBRARY_PATH=./lib ADSP_LIBRARY_PATH=./lib ./bin/llama-server --device HTP0 ### 3. Catalog entries Add three new runtimes to docs/runtime-catalog.json under Ericson246/llama.cpp: - android-arm64-hexagon (priority: higher than cpu, same tier as vulkan) - windows-arm64-hexagon (priority: same as vulkan) - linux-arm64-hexagon (priority: same as vulkan) ### 4. Backend detection priority update The selection engine needs: - Android: Hexagon NPU > Vulkan > CPU - Windows ARM: Hexagon NPU > Vulkan > CPU - Linux ARM: Hexagon NPU > CPU (no Vulkan OpenCL on Linux Snapdragon) ## Build infrastructure ### Android - Docker image: ghcr.io/snapdragon-toolchain/arm64-android:v0.7 - GitHub Actions runner: ubuntu-latest (runs Docker) - Build time: ~15-30 min first build, ~3-8 min incremental - Includes: Android NDK r28b, Hexagon SDK 6.6.0.0, OpenCL SDK ### Windows ARM64 - Requires: ARM64 Windows runner (GitHub doesn't provide, need self-hosted or cross-compile) - Hexagon SDK 6.6+ (free w/ Qualcomm account) - OpenCL SDK 2.3+ - MSVC 2026 arm64 toolchain - Driver signing certificate ### Linux ARM64 - Docker image: ghcr.io/snapdragon-toolchain/arm64-linux:v0.1 - Host: Linux x86_64 runner - Cross-compilation only ## Known limitations | Limitation | Impact | |------------|--------| | ggml-hexagon is experimental | May have stability issues | | Only Q4_0, Q8_0, MXFP4, FP32 quantizations | Most HF models use Q4_K_M ├ö├ç├ need re-quantization or upstream support | | No Flash Attention in prefill | Falls back to CPU for prefill FA | | No on-disk REPACK format | ~600ms extra load time per model | | Windows driver signing required | Test mode (self-signed) or production (EV cert) | | Multi-device instability on Windows | Intermittent hangs with GGML_HEXAGON_NDEV>1 | | Models >8B need multiple HTP sessions | 2 sessions for 8B, 4 for 20B; may exhaust NPU memory | ## Future considerations - If upstream merges the QNN backend (PR #12063), switch to -DGGML_QNN=ON for better op support and stability - QNN SDK approach supports more quantization types and doesn't need driver signing on Windows - Snapdragon X2 Elite (80 TOPS) expected 2026 ├ö├ç├ may require additional Hexagon v82+ arch support ## Related resources - llama.cpp Snapdragon docs - PR #16547 ├ö├ç├ ggml-hexagon backend - PR #21705 ├ö├ç├ Improved op queuing - PR #19150 ├ö├ç├ Windows support - Snapdragon toolchain Docker - chraac/llama-cpp-qnn-builder - Qualcomm Hexagon SDK (free) - Qualcomm QNN SDK (free) - SpecPicks benchmarks ├ö├ç├ Snapdragon X Elite NPU - llama.cpp on Linux Snapdragon discussion