Skip to content

hex-dma: introduce direct-mapped DMA cache that is better suited for HVX FA mask handling - #29282

Merged
max-krasnyansky merged 1 commit into
ggml-org:masterfrom
qualcomm:hexagon-dma-cache
Sep 22, 2026
Merged

max-krasnyansky merged 1 commit into
ggml-org:masterfrom
qualcomm:hexagon-dma-cache

Conversation

@max-krasnyansky

Copy link
Copy Markdown
Member

Overview

Full LRU cache is an overkill for HVX FA loops.
This PR introduces simple direct mapped cache and updates FA to use that in the HVX paths.
The original LRU cache has been renamed to dma_cache_fa (fully-assoc).

Address issue identified in #29210

Additional information

Some perf numbers from IQ9.

./scripts/snapdragon/run.py --target  ubuntu:maxk@192.168.1.87 --devices HTP0 -- llama-bench -m ../gguf/Llama-3.2-1B-Instruct-Q4_0.gguf -t 6 -ngl 99 --ubatch-size 1024 -p 0 -d 0,2048,4096,8192 -n 64
+ ssh maxk@192.168.1.87 cd ~/llama.cpp && ulimit -c unlimited && LD_LIBRARY_PATH=./lib ADSP_LIBRARY_PATH=./lib GGML_HEXAGON_DEVICES=HTP0 GGML_HEXAGON_OPPOLL=1 ./bin/llama-bench -m ../gguf/Llama-3.2-1B-Instruct-Q4_0.gguf -t 6 -ngl 99 --ubatch-size 1024 -p 0 -d 0,2048,4096,8192 -n 64 --device HTP0
ggml-hex: Loading driver libcdsprpc.so
ggml-hex: FASTRPC_GET_DOMAINS query failed (0x6c), using static CDSP domains
ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
ggml-hex: Hexagon Arch version v73, DMA64 disabled
ggml-hex: device 0: HTP0 (phys=0, virt=0, domain=cdsp:3)
| model          |       size |  params | ngl | threads | n_ubatch | dev   |         test | master    t/s | this-PR   t/s |
| -------------- | ---------: | ------: | --: | ------: | -------: | ----- | -----------: | ------------: | ------------: |
| llama 1B Q4_0  | 727.75 MiB |  1.24 B |  99 |       6 |     1024 | HTP0  |         tg64 |  48.13 ± 0.06 |  50.86 ± 0.10 | 
| llama 1B Q4_0  | 727.75 MiB |  1.24 B |  99 |       6 |     1024 | HTP0  | tg64 @ d2048 |  29.52 ± 0.03 |  32.23 ± 0.02 | 
| llama 1B Q4_0  | 727.75 MiB |  1.24 B |  99 |       6 |     1024 | HTP0  | tg64 @ d4096 |  22.15 ± 0.02 |  24.43 ± 0.01 | 
| llama 1B Q4_0  | 727.75 MiB |  1.24 B |  99 |       6 |     1024 | HTP0  | tg64 @ d8192 |  14.77 ± 0.00 |  16.49 ± 0.00 | 

Requirements

@max-krasnyansky
max-krasnyansky requested a review from a team as a code owner September 22, 2026 18:09
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning Hexagon labels Sep 22, 2026
@max-krasnyansky

Copy link
Copy Markdown
Member Author

@lhez for review and ack

@max-krasnyansky max-krasnyansky added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Sep 22, 2026
@max-krasnyansky
max-krasnyansky merged commit e6ab7c1 into ggml-org:master Sep 22, 2026
18 of 19 checks passed
@BrewTestBot BrewTestBot mentioned this pull request Sep 23, 2026
1 task done
LadislavSopko pushed a commit to 0ics-srls/llama.cpp that referenced this pull request Oct 5, 2026
frostyautumnleaf pushed a commit to frostyautumnleaf/llama.cpp that referenced this pull request Oct 5, 2026
edwardyoon pushed a commit to edwardyoon/focus-llama that referenced this pull request Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Hexagon merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants