CPU only infrastructure related observations

[arunaagrawal]

While using a vCPU 20 cores and 128 GB RAM on a CPU only architecture we observe:

1. 50% of CPUs are used at max. But may be this problem is coming from llamacpp side as we observed same there as well.
2. When using parallel with 3 instance using web ui. The 3 answers starts in parallel to start with which is great but invariable the 3rd answer gets confused and starts going into some kind of loop spitting same 1-2 words over and over again.

[Djip007]

50% of CPUs are used at max. But may be this problem is coming from llamacpp side as we observed same there as well.

You can add -t 20 for use of all thread, by default on hardware x86 it use only core and remove hyperthreading (si 1/2 of declared thread). I don't know how it get for VirtualCPU, nor for non hyperthreading CPU.

Note: on zen3/4 it is best to use only 1 thread par core (and I thing on many other CPU) in most case.

[arunaagrawal]

Thanks, I will give it a try and I looked into my previous llamacpp tests and it does have a -t 20 in there.

Any comments on the 2nd issue pls.

[colindaven]

I get the word repeating problem using the WebUI on a remote cpu only server too. See #573