Extremely slow response time from Prompts

[Bigcow11]

Has anyone else had issues with the response taking forever on alpaca? Have the 13B version installed and operational; however, when prompted for an output the response is extremely slow.

For example: 5+ minutes to output text responding to the sample question of: "What are lists in python?". This is also apparent through terminal so not isolated to the web GUI.

Hardware is more than sufficient as well to run it

[DartPower]

Also have this issue, but IDK this is normal or not.

[Bigcow11]

Also have this issue, but IDK this is normal or not.

I don't believe its normal, on the readme page he includes the .gif of the speed which matches chat gtp3 speed

[freezah]

Same here. Running on 16 Core AMD CPU and is extremely slow.

[trevtravtrev]

What hardware? Likely the only issue I can think of. I'm running 7B, 13B, and 30B on 32GB RAM and a beefy cpu. 7B is snappy, 13B is still fast, 30B takes a couple minutes to output the full answers.

[freezah]

@trevtravtrev
I’m using AMD 5950X (32 threads) with 128GB of RAM. Tried with 7B as well and also super slow. Even I run the “chat” with -t 32 parameter, I see mostly one thread being used at 100% and rest is kind of idling.

[pnadj]

Same with the basic 7B model

[ninp0]

+1 debian VM, 16 core & 64 gb memory
6.1.0-kali5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.12-1kali2 (2023-02-23) x86_64 GNU/Linux

alpaca.cpp $ ./chat -t 16 -m ggml-alpaca-7b-q4.bin --interactive-start
main: seed = 1679691725
llama_model_load: loading model from 'ggml-alpaca-7b-q4.bin' - please wait ...
llama_model_load: ggml ctx size = 6065.34 MB
llama_model_load: memory_size =  2048.00 MB, n_mem = 65536
llama_model_load: loading model part 1/1 from 'ggml-alpaca-7b-q4.bin'
llama_model_load: .................................... done
llama_model_load: model size =  4017.27 MB / num tensors = 291

system_info: n_threads = 16 / 16 | AVX = 1 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
main: interactive mode on.
sampling parameters: temp = 0.100000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000


== Running in chat mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMA.
 - If you want to submit another line, end your input in '\'.
>

[KaruroChori]

@freezah
I would suggest you using 30 threads, not 32. It is better to keep a few available for other processes, based on your system load.

[freezah]

@KaruroChori
I totally agree. I used 32 just to verify if the application will work better than with default 4 threads.

I just run it with 30 threads and observing same issue. Even though I see 30 forked processes in 'htop', only one of them consumes 100% of single thread, rest is consuming 3.3-4.0% on average.

Model loading is quick, but now it's already 10 minutes and the prompt still didn't show up:

llama_model_load: loading model from './ggml-alpaca-7b-q4.bin' - please wait ...
llama_model_load: ggml ctx size = 6065.34 MB
llama_model_load: memory_size = 2048.00 MB, n_mem = 65536
llama_model_load: loading model part 1/1 from './ggml-alpaca-7b-q4.bin'
llama_model_load: .................................... done
llama_model_load: model size = 4017.27 MB / num tensors = 291

system_info: n_threads = 30 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
main: interactive mode on.
sampling parameters: temp = 0.100000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000

== Running in chat mode. ==

• Press Ctrl+C to interject at any time.
• Press Return to return control to LLaMA.
• If you want to submit another line, end your input in ''.

[Bigcow11]

What hardware? Likely the only issue I can think of. I'm running 7B, 13B, and 30B on 32GB RAM and a beefy cpu. 7B is snappy, 13B is still fast, 30B takes a couple minutes to output the full answers.

7B runs slow too, I have a 24 thread with 128gb of ram and dedicated gpus so it’s not the hardware bottle necking. There’s an issue with model

[Bigcow11]

Have we found any solutions or successful work arounds?

[KaruroChori]

I tested both the 7b and the 30b on a xeon 2650Lv4 128GB of ram and a ryzen 5950x with 64GB of ram. Performance are comparable between the two platform (between 2-3 times faster for the ryzen) and usable.
I am not sure why some people are getting an extremely slow generation.
When the process is running, all cores should be around 100%, not just one.

[freezah]

@KaruroChori

Which host system distro, version, kernel you are using ?

[KaruroChori]

My uname -a:

Linux [redacted] 5.15.0-40-generic #43-Ubuntu SMP Wed Jun 15 12:54:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Linux [redacted] 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[kaiserfr]

Same here.
Have an AMD A10-6700 CPU and 32GB RAM. I try to run it on Ubuntu Ubuntu 20.04.6 and WSL2.
My guess is that some speed optimizations like AVX or NEON do not work with AMD processors.

Do AMD processers have other, similar speedup flags, which must be used instead?