Confirmed now, q4_0 is (still) broken

[GameOverFlowChart]

So I finally had the chance to make the comparison. Since the update to q4_0 which should make them more performant, they became unuseful slow. I made a comparison with q4ks.
8B q4_0 4.67GB: freezes not just the app but the whole phone.
8B q4_K_S 4.68GB: runs like always. Notice that it's even minimal bigger in file size.
Reminder: I was able to run Q4_0_4_4 in the past and with this version q4_0 should behave like Q4_0_4_4.

I also made sure that the problem is not related to having the weights in internal storage or external. The behavior is the same in both cases.

I made the test with version 0.8.4a.
If the problem is in llamacpp and not chatterui, we should probably tell them about the problem.

[Vali-98]

Thanks for this update, there actually was a recent bug found in llama.rn on some fp16 compilation flags causing issues,

I uploaded a v0.8.5 beta with an updated llama.cpp / build process. Let me know if this works.

[GameOverFlowChart]

Thanks for this update, there actually was a recent bug found in llama.rn on some fp16 compilation flags causing issues,

I uploaded a v0.8.5 beta with an updated llama.cpp / build process. Let me know if this works.

Still the same, no problem with q4ks, but q4_0 slows down the device (the phone kills my dynamic wallpaper, I guess to free some resources also the Android buttons respond late so it's not easy to leave the app). But the loading bar filled fast this time, at first I thought the problem was solved.

[Vali-98]

I'm assuming something is going very wrong with the model repacking, or for whatever reason memory is not being released properly.

What hardware was this tested on? Does your device allow for 'memory extension' using the device storage? If so, is it enabled?

[GameOverFlowChart]

I'm assuming something is going very wrong with the model repacking, or for whatever reason memory is not being released properly.

What hardware was this tested on? Does your device allow for 'memory extension' using the device storage? If so, is it enabled?

12gb ram, Snapdragon 888, Android 14, memory extension is disabled and I wouldn't activate it ever.

[GameOverFlowChart]

Does chatterui or llamacpp do the detection of supported cpu instructions? What if the model gets repacked as Q4_0_4_8 instead of Q4_0_4_4?
The old version of chatterui was able to correctly show which one was supported by my device. According to my memory despite being slow I had the model outputing a response as I first encountered this problem. I don't know if that would be possible with wrong repacking or if it would just be as inefficient as it is.

[Vali-98]

llama.rn manages the cpu flag selection by checking flags provided by the OS. For i8mm, it simply checks for the i8mm flag.

If you have logcat, you can filter rnllama to check the startup routine, it should list your device capabilities:

[3-ark]

I think that's because of llamacpp. They cease to support that quant from an update. I can't remember exactly, but you can search there.

[Vali-98]

I think that's because of llamacpp. They cease to support that quant from an update. I can't remember exactly, but you can search there.

This issue is more about Q4_0 vs the now outdated Q4_0_4_8 quants. For some reason, performance of Q4_0 is worse.