QNN NPU image models and mtp fix on s24U

[jnk333]

Hi Jegly, I find you app amazing and its doing so much for the android local ai community so far, i use the app on my s24u with bypass charging and a cooler but im getting less than 10 tps when running gemma 4 e4b on gpu and ik its a bug because its higher than that in the ai edge gellery ap and off grid, and mtp just doesnt work at all and doesnt increase my tps, new thinking only gguf models like lfm 2.5 8b a1b are also bugged in the sense that when you toggle thinking on it still thinks silently and doesnt show it's thinking, I would please like more advance image gen settings and text gen settings to be added, plus i wanted to please ask for you to try to eventually get the image gen side of Box to be like local dream allowing for QNN npu models to be used since they are very effecient and fast and most of the big sd 1,5 and sdxl models have been converted to QNN NPU compatible formats that i currently comfortably run on my s24 you, if rag improves, tool calling is better, mtp works, there is npu image models with with pro sumer settings and the allowance to load sd 1.5 NPU models and small text models at the same time you app will be the undisputed goat of local ai on android, thank for what youve done already have a nice day.

[jnk333]

i forgot to mention that benchmarking for gguf models isnt possible yet sadly like in pocketpal at least for me and i wish for it to be added and lastly that the Box app has the chance be the first app on mobile to add mtp to gguf models and if you do it will be absolutely legendary, but please take your your time to develop the app youve already done so much

[jnk333]

in the ai edge gallery on my s24u with cooling and bypass charging i get 17 decode and 33+ with mtp on gemma 4 e4b, in Box Mtp doesnt work and i get rougly 7-9 tps, gemma 4 e2b gets 33 decode in the ai edge gallery and 60+ with mtp, its roughly 15 tps in box, and mtp is not working i did lots of reinstalls, cache clears, model redownload, perfectly cooled the phone, cleared ram too and the s24 u has 12gb ram so IK for sure its not a low memory issue

[jegly]

@jnk333

Thanks for the detailed feedback. Below are updates on each of the points you
raised:

Speed on Gemma 3n E4B (GPU, S24 Ultra) Gemma 3n models currently only support
NPU acceleration on MediaTek chipsets. On the S24 Ultra (Snapdragon sm8650), the
E4B model runs on the GPU. Because E4B is a large model, its decode phase is
memory-bandwidth bound, and the S24 Ultra is prone to thermal throttling under
sustained loads. For optimal performance on your Snapdragon device, i recommend
using the Gemma 3 1B (NPU) model, which is optimized specifically for the sm8650
SoC.

Comparison to Other Apps Box is deliberately pinned to a specific LiteRT runtime
version to prevent GPU inference regressions introduced in newer releases.
Performance variations compared to other applications are expected due to this
stability trade-off. If you can share the results from the in-app Benchmark
screen, i can investigate the performance metrics more closely.

MTP / Speculative Decoding This is a limitation of the current public runtime
API, which does not expose the tuning parameters required for Multi-Token
Prediction (MTP) to accelerate decoding. Consequently, enabling this feature
will not increase token-per-second rates at this time. Additionally, please note
that Box does not utilize GPU acceleration for the LLaMA backend (where MTP
improvements would typically be observed). Box uses LiteRT as its primary
engine, while LLaMA is provided as a secondary compatibility fallback.

"Thinking" Tags on Imported GGUF Models (e.g., LFM 2.5) Different models use
varying tags to mark reasoning tokens, and our GGUF import parser does not yet
support all variations. i will look into expanding tag compatibility, but
please note that GGUF import is intended as a convenience fallback. LiteRT
remains our primary development focus, so this issue is currently a lower
priority.

Advanced Image and Text Generation Settings i plan to expand image-generation
options (including additional samplers and larger models) in future updates.
Text-generation settings like temperature, top-K, and top-P are already
configurable in the model settings. Let us know if there are other specific
parameters you would like to see, though development timelines will depend on
our current bandwidth.

QNN NPU Image Models i do not plan to support QNN-specific image models.
Supporting QNN requires a separate SDK, model format, and native layer, which
does not align with Box's offline-first, cross-platform architecture.

RAG and Tool Calling RAG and tool-calling improvements are on our long-term
roadmap. Tool calling is currently supported via MCP.

Thank you again for using Box and sharing your feedback.

[jnk333]

These are all the benchmarks I did on my s24 Ultra with bypass charging and cooling, some of these scores might be 1 token lower due to not having performance mode turned on, the screenshot with my homescreen is of the npu model you suggested i use crashing box, the blue ui is the Ai edge gallery and the orange ones are the Box app, e2b seems to have the biggest performance fall off in the group when using it in box, performing better than e4b by only 6 tokens, I under the cross-platform argument for the Npu image models but the very fact that there already is care for Google Tpu models, snapdragon npu models and mediatek npu model already means not everything will be cross-platform in the first place, the npu image models already have been converted and just need the app to be able to run them but I aware it's hard work so it's fine if it doesn't make it, if you see the app versions vary it's due to 1.0.13 being the version of the ai edge gallery that introduced mtp and being the one that can still use it on my s24u, the newer versions are bugged sadly and don't improve speed all the models were benchmarked with same settings nonetheless so the results are very accurate for my case (very little thermal constraints), thank you for all you've done and I hope these benchmarks help, lfm 2.5 8B A1B also gets 20 decode on my phone running on cpu, tested across multiple apps. I also just added a chart so you have a easier time seeing everything

[jnk333]

I'm hoping a chat history feature like the one in the new ai edge gallery versions can be added, it helps isolate conversations, save memory and stay organized, it's a absolute shame the npu mode can't be used in box let alone be used in regular chat mode in the ai edge gallery app

[jegly]

@jnk333 .. you provided screenshots with no context.. it clearly shows npu? is that box 3.0.0 or upstream ai edge gallery ? Using clear English helps thanks.

[jnk333]

Like I said the orange screenshots are Box and blue is the ai edge gallery and if you look at the blue and orange screenshots it shows the version of box or ai edge gallery app the benchmark was done on, like I said npu does not work in box for my phone but gpu works for the same model, npu does work in the ai edge gallery but not box

[jegly]

Thanks for the kind words and the detailed report — let me go through each point honestly, since a few of these are known and
a couple are genuinely useful.

The NPU / 2000 tok/s you're seeing in AI Edge Gallery: that's Gallery's per-SoC NPU build for your S24U (Snapdragon 8 Gen 3 /
SM8650), where it ships exactly one Qualcomm NPU driver. Box currently bundles all the vendor NPU drivers in one APK, which
makes the wrong one load on Snapdragon and crash — so on your phone Box can't yet reach the NPU and falls back to GPU. I'm
actively reworking this into per-vendor builds (a Qualcomm build that ships only the Snapdragon driver, exactly like Gallery
does), which is what unlocks that NPU path on the S24U. That's the big speedup you're after.

<10 tok/s on Gemma 4 E4B (GPU): worth separating two things — E4B (the 3n E4B multimodal model) has no NPU build anywhere,
including Gallery and Off Grid, so for E4B specifically everyone is on GPU/CPU; the NPU gains will show up on the 1B / E2B
builds once the Snapdragon NPU path above lands. That said, <10 tps on an 8 Gen 3 GPU does sound low — can you tell me the
exact model file you loaded and confirm the accelerator was set to GPU (not CPU)? I'd like to compare apples-to-apples
against Gallery's E4B number.

MTP / speculative decoding "doesn't work": that's a known limitation, not specific to your device. MTP engages on models that
support it, but the current runtime doesn't expose the draft-length control, so there's no real decode speedup to show yet —
on any phone. It's on the list, but it's gated on an upstream runtime change.

Thinking models showing silent reasoning with no visible output: that's a rendering gap on the new reasoning channel — the
model is producing the "thought" stream but it isn't being surfaced. I'll look at getting that displayed.

QNN NPU image generation (Local Dream style, SD 1.5 / SDXL in QNN format): Box's image gen today runs on sd.cpp (CPU/GPU),
which is a completely separate pipeline from QNN NPU. It's a genuinely interesting direction and it's on the longer-term
list, but it's a sizable piece of work — not near-term. Advanced image/text generation settings are easier and I can expand
those sooner.

Appreciate you pushing on this — the per-vendor NPU rework is the headline item and it's in progress. I'll reference this
issue when the Snapdragon NPU build is ready to test, if you're up for trying it on the S24U.

[jnk333]

Thanks for the kind words and the detailed report — let me go through each point honestly, since a few of these are known and
a couple are genuinely useful.

The NPU / 2000 tok/s you're seeing in AI Edge Gallery: that's Gallery's per-SoC NPU build for your S24U (Snapdragon 8 Gen 3 /
SM8650), where it ships exactly one Qualcomm NPU driver. Box currently bundles all the vendor NPU drivers in one APK, which
makes the wrong one load on Snapdragon and crash — so on your phone Box can't yet reach the NPU and falls back to GPU. I'm
actively reworking this into per-vendor builds (a Qualcomm build that ships only the Snapdragon driver, exactly like Gallery
does), which is what unlocks that NPU path on the S24U. That's the big speedup you're after.

<10 tok/s on Gemma 4 E4B (GPU): worth separating two things — E4B (the 3n E4B multimodal model) has no NPU build anywhere,
including Gallery and Off Grid, so for E4B specifically everyone is on GPU/CPU; the NPU gains will show up on the 1B / E2B
builds once the Snapdragon NPU path above lands. That said, <10 tps on an 8 Gen 3 GPU does sound low — can you tell me the
exact model file you loaded and confirm the accelerator was set to GPU (not CPU)? I'd like to compare apples-to-apples
against Gallery's E4B number.

MTP / speculative decoding "doesn't work": that's a known limitation, not specific to your device. MTP engages on models that
support it, but the current runtime doesn't expose the draft-length control, so there's no real decode speedup to show yet —
on any phone. It's on the list, but it's gated on an upstream runtime change.

Thinking models showing silent reasoning with no visible output: that's a rendering gap on the new reasoning channel — the
model is producing the "thought" stream but it isn't being surfaced. I'll look at getting that displayed.

QNN NPU image generation (Local Dream style, SD 1.5 / SDXL in QNN format): Box's image gen today runs on sd.cpp (CPU/GPU),
which is a completely separate pipeline from QNN NPU. It's a genuinely interesting direction and it's on the longer-term
list, but it's a sizable piece of work — not near-term. Advanced image/text generation settings are easier and I can expand
those sooner.

Appreciate you pushing on this — the per-vendor NPU rework is the headline item and it's in progress. I'll reference this
issue when the Snapdragon NPU build is ready to test, if you're up for trying it on the S24U.

I don't mind testing the build when it arrives, the screenshots i provided also show which that everything was run on gpu, there was a typo, I was referring to the 1b npu model (hate that it's only available in prompt mode), I used the gemma 4 e2b and gemma 4 e4b models provided in box and the ai edge gallery, getting 11 tps is one thing on the s24u is one thing but the strangest occurrence was the gemma 4 e2b being so slow on box vs the ai edge gallery, regardless thank you so much for you timely responses and your work this app is going to be pioneer of mobile local ai and I know it!

[jegly]

@jnk333 can you test a APK in about 10 minutes ?

[jnk333]

@jnk333 can you test a APK in about 10 minutes ?

Sure, everything is ready just let me know what to specifically test

[jegly]

@jnk333 give this a try https://github.com/jegly/Box/releases/tag/experimental