falcon : speed-up prompt processing

[ggerganov]

The performance of Falcon 7B should be comparable to LLaMA 7B since the computation graph is computationally very similar.

Here are the current numbers on M2 Ultra for LLaMA, LLaMA-v2 and Falcon 7B:

../scripts/run-all-perf.sh ${model} "f16 q8_0 q4_0"

model	size	params	backend	ngl	test	t/s
LLaMA 7B mostly F16	12.55 GiB	6.74 B	Metal	999	pp 512	665.95 ± 0.18
LLaMA 7B mostly Q8_0	6.64 GiB	6.74 B	Metal	999	pp 512	630.28 ± 0.16
LLaMA 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	999	pp 512	632.32 ± 0.22
LLaMA 7B mostly F16	12.55 GiB	6.74 B	Metal	999	tg 64	29.73 ± 0.01
LLaMA 7B mostly Q8_0	6.64 GiB	6.74 B	Metal	999	tg 64	61.47 ± 0.06
LLaMA 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	999	tg 64	86.96 ± 0.08

build: dd0dc36 (1100)

model	size	params	backend	ngl	test	t/s
llama2 7B mostly F16	12.55 GiB	6.74 B	Metal	999	pp 512	666.12 ± 0.10
llama2 7B mostly Q8_0	6.64 GiB	6.74 B	Metal	999	pp 512	630.21 ± 0.20
llama2 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	999	pp 512	632.32 ± 0.17
llama2 7B mostly F16	12.55 GiB	6.74 B	Metal	999	tg 64	29.74 ± 0.02
llama2 7B mostly Q8_0	6.64 GiB	6.74 B	Metal	999	tg 64	61.55 ± 0.04
llama2 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	999	tg 64	86.88 ± 0.08

build: dd0dc36 (1100)

model	size	params	backend	ngl	test	t/s
Falcon 7B mostly F16	13.44 GiB	7.22 B	Metal	999	pp 512	403.68 ± 1.27
Falcon 7B mostly Q8_0	7.14 GiB	7.22 B	Metal	999	pp 512	390.41 ± 1.77
Falcon 7B mostly Q4_0	3.92 GiB	7.22 B	Metal	999	pp 512	390.94 ± 1.75
Falcon 7B mostly F16	13.44 GiB	7.22 B	Metal	999	tg 64	29.47 ± 0.01
Falcon 7B mostly Q8_0	7.14 GiB	7.22 B	Metal	999	tg 64	60.01 ± 0.05
Falcon 7B mostly Q4_0	3.92 GiB	7.22 B	Metal	999	tg 64	86.07 ± 0.02

build: 611363a (1110)

Although the Text Generation speed for Falcon is comparable to LLaMA, I observe a significant performance drop in the Prompt Processing task. This is on M2 Ultra with Metal, but I think last time I checked, the CUDA performance experiences a similar drop.

Hypothesis

I haven't profiled the run yet, but I suspect the cause is in the concatenated QKV matrix multiplication (MM):

https://github.com/ggerganov/llama.cpp/blob/dd0dc366dab10e8df28d3924e7f313b5c695e908/llama.cpp#L2634-L2638

For some reason, this is probably slower compared to what we do in LLaMA, where we have separated the QKV tensor into 3 individual Q, K and V tensors:

https://github.com/ggerganov/llama.cpp/blob/dd0dc366dab10e8df28d3924e7f313b5c695e908/llama.cpp#L2292-L2306

We should either speed-up the current QKV implementation, or change the convert script to output:

• LLM_TENSOR_ATTN_Q
• LLM_TENSOR_ATTN_K
• LLM_TENSOR_ATTN_V

instead of:

• LLM_TENSOR_ATTN_QKV

https://github.com/ggerganov/llama.cpp/blob/dd0dc366dab10e8df28d3924e7f313b5c695e908/convert-falcon-hf-to-gguf.py#L220-L239

My intuition is that if this is indeed the reason for the slow-down in Falcon, the combined QKV approach, if optimized correctly, should yield better performance than the other since we do one single MM instead of 3 separate MMs. So we should also consider switching LLaMA graph if this turns out to be the case and we are able to optimize it and the improvement is significant

Edit: some more results with CUDA on RTX 4080

../scripts/run-all-perf.sh ${model} "f16 q8_0 q4_0" "-ngl 999 -t 1 -n 64"

model	size	params	backend	ngl	threads	test	t/s
llama2 7B mostly F16	12.55 GiB	6.74 B	CUDA	999	1	pp 512	2122.08 ± 0.58
llama2 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	999	1	pp 512	3343.20 ± 9.31
llama2 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	999	1	pp 512	3439.35 ± 10.13
llama2 7B mostly F16	12.55 GiB	6.74 B	CUDA	999	1	tg 64	45.95 ± 0.01
llama2 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	999	1	tg 64	77.85 ± 0.01
llama2 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	999	1	tg 64	130.43 ± 0.01

build: 611363a (1110)

model	size	params	backend	ngl	threads	test	t/s
Falcon 7B mostly F16	13.44 GiB	7.22 B	CUDA	999	1	pp 512	1885.04 ± 2.29
Falcon 7B mostly Q8_0	7.14 GiB	7.22 B	CUDA	999	1	pp 512	2849.60 ± 5.96
Falcon 7B mostly Q4_0	3.92 GiB	7.22 B	CUDA	999	1	pp 512	2754.78 ± 6.54
Falcon 7B mostly F16	13.44 GiB	7.22 B	CUDA	999	1	tg 64	36.07 ± 0.01
Falcon 7B mostly Q8_0	7.14 GiB	7.22 B	CUDA	999	1	tg 64	54.68 ± 0.03
Falcon 7B mostly Q4_0	3.92 GiB	7.22 B	CUDA	999	1	tg 64	75.76 ± 0.01

build: 611363a (1110)

[akawrykow]

@ggerganov I'm a total noob in this space but have been lurking the issues and PR for fun to try and learn something.

You mentioned:

My intuition is that if this is indeed the reason for the slow-down in Falcon, the combined QKV approach, if optimized correctly, should yield better performance than the other since we do one single MM instead of 3 separate MMs

Could you clarify the 'if optimized' part? Do you have any ideas in mind for this? Do you mean how we are constructing the wqkv tensor to begin with in convert-falcon-hf-to-gguf.py or do you mean some aspect of ggml_mul_mat is slow when it takes in a tensor of this shape?

Intuitively, I understand that 3 MMs might be slower than a single MM but I'm not sure where to go from there. I also intuitively understand that we might want to try keeping them as separate tensors like we have in Llama (and as a noobie this seems like something that would be easier for me to try). Can you share the commands you ran to generate the performance charts above?

Thanks!

[ggerganov]

I just ran a few more tests with CUDA and it seems the TG task is also slower for Falcon compared to LLaMA.

Could you clarify the 'if optimized' part?

I guess the implemented matrix multiplication kernels are fine-tuned for LLaMA-specific shapes and when we run with non-LLaMA models, we probably don't get the best performance. But this is just a guess. We should investigate either tweaking the kernels (i.e. block sizes, threads per block, etc.) or re-arranging the weight data in a similar way. For example, I think the convert-falcon-hf-to-gguf.py script can be easily modified to output the Q, K, V tensors as we do with LLaMA and see if this helps.

Running a single MM instead of 3 would reduce the number of nodes in the graph and hence we will have less synchronization points. Probably not a big win, but might be measurable, especially for smaller models.

To get these tables, I build the project with CMake in build folder and then run the following:

# model structure
ls -l ../models
drwxr-xr-x  13 416 Aug 27 14:36 falcon-7b
drwxr-xr-x  17 544 Aug 27 14:42 llama-7b
drwxr-xr-x  18 576 Aug 26 23:29 llama-7b-v2

# generate quantum models (run once)
../scripts/qnt-all.sh llama-7b    "q8_0 q4_1 q4_0"
../scripts/qnt-all.sh llama-7b-v2 "q8_0 q4_1 q4_0"
../scripts/qnt-all.sh falcon-7b   "q8_0 q4_1 q4_0"

# run `llama-bench`
../scripts/run-all-perf.sh llama-7b    "f16 q8_0 q4_0"
../scripts/run-all-perf.sh llama-7b-v2 "f16 q8_0 q4_0"
../scripts/run-all-perf.sh falcon-7b   "f16 q8_0 q4_0"

One can also try to profile the computation on the CPU using (see GGML Tips & Tricks):

# build with perf stats enabled
LLAMA_PERF=1 make -j

# run LLaMA-v2 prompt processing with 512 tokens
./main -m models/llama-7b-v2/ggml-model-q4_0.gguf -p "$(printf "%0.s hello" {1..510})" -c 2048 -n 1 -s 1 -t 8 --no-mmap 

# run Falcon prompt processing with 512 tokens
./main -m models/falcon-7b/ggml-model-q4_0.gguf -p "$(printf "%0.s hello" {1..512})" -c 2048 -n 1 -s 1 -t 8 --no-mmap

Or also look at the graph plots and see if we doing some unnecessary computations:

LLaMA graph

Falcon graph