optim on tinyblas #654

Djip007 · 2024-12-08T02:57:03Z

Djip007
Dec 8, 2024

ikawrakow/ik_llama.cpp#71 have a good idea.

I'll figure to add it in tinyblas and id work great. (and I add quant in FP16/BF16 in all case for B to reduce memory bandwidth. work nice for AVX512/AVX2 kernel)

https://github.com/Djip007/llama.cpp/blob/perfo/tinyblas/ggml/src/ggml-cpu/llamafile/sgemm.cpp#L297

Djip007 · 2024-12-08T03:12:28Z

Djip007
Dec 8, 2024
Author

[edit: 2024/12/09]: OK some correction and more tuning
some bench with Mistral-Nemo-Instruct-2407:

on ryzen 5950x (zen3) I get (from llama.cpp code):

with llamafile 0.8.17 vs this code

cpu_info	model_filename	test	0.8.17 t/s	new code
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp1	2.19	2.21
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp2	4.37	4.31
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp3	6.46	6.44
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp4	4.97	8.58
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp5	6.12	10.71
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp6	12.90	12.78
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp7	8.70	14.86
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp8	9.81	16.92
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp9	19.19	18.93
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp10	12.49	20.92
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp11	13.57	23.01
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp12	25.43	24.89
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp13	16.22	26.94
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp14	17.26	28.78
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp15	31.09	30.64
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp16	19.98	32.45
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp30	34.87	42.87
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp32	28.64	43.14
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp64	31.97	45.05
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp65	32.18	44.94
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp120	36.41	46.27
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp128	34.18	46.11
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp130	33.91	45.93
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp240	35.88	46.50
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp255	35.81	46.54
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp256	34.69	46.40
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp510	34.54	46.01
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp512	33.64	45.81
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp1024	33.36	45.05
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp1025	32.15	44.18
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp2048	32.76	43.68
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	tg128	2.19	2.21

cpu_info	model_filename	test	0.8.17 t/s	new code
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp1	2.19	2.19
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp2	4.37	4.28
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp3	6.44	6.41
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp4	4.90	8.53
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp5	6.19	10.64
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp6	12.83	12.71
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp7	8.54	14.81
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp8	9.85	16.88
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp9	19.10	18.90
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp10	12.24	20.88
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp11	13.53	22.94
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp12	25.38	24.89
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp13	16.00	26.87
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp14	17.27	28.74
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp15	31.60	30.66
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp16	19.67	32.55
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp30	39.28	54.23
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp32	30.30	55.37
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp64	34.77	58.46
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp65	35.08	57.47
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp120	40.33	58.01
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp128	37.34	57.76
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp130	37.36	58.28
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp240	40.43	58.66
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp255	40.45	58.62
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp256	38.88	58.57
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp510	39.67	57.94
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp512	38.94	57.51
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp1024	38.42	56.41
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp1025	37.90	53.96
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp2048	37.56	54.93
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	tg128	2.19	2.19

Not the best that we can have with this CPU but we may need a true BLIS kernel for best (I think we can have ~80 t/s)
But that a "simple" change

on AMD Ryzen™ 9 7940HS (zen4)

with llamafile 0.8.17 vs this

cpu_info	model_filename	test	0.8.17 t/s	new code
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp1	2.50	2.51
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp2	4.98	4.74
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp3	4.02	7.23
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp4	9.91	9.91
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp5	12.33	12.37
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp6	7.91	14.78
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp7	9.26	17.09
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp8	7.30	19.41
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp9	11.78	21.63
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp10	24.02	24.02
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp11	14.41	26.30
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp12	15.67	28.43
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp13	11.70	29.41
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp14	18.10	31.56
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp15	34.31	33.51
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp16	20.57	35.87
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp30	50.37	51.09
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp32	36.26	51.04
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp64	50.78	57.23
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp65	54.97	57.33
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp120	61.28	65.51
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp128	47.93	65.81
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp130	61.80	66.31
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp240	64.80	70.40
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp255	64.09	70.32
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp256	60.79	70.34
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp510	59.59	68.26
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp512	56.96	67.97
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp1024	56.23	66.76
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp1025	55.21	64.84
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp2048	54.72	63.96
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	tg128	2.50	2.52

cpu_info	model_filename	test	0.8.17 t/s	new code
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp1	2.50	2.50
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp2	4.99	4.81
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp3	4.00	7.19
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp4	9.81	9.82
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp5	12.18	12.25
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp6	7.92	14.58
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp7	9.18	16.52
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp8	7.26	18.76
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp9	11.65	20.86
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp10	23.25	22.80
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp11	14.25	24.90
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp12	15.52	26.70
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp13	11.63	26.99
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp14	17.87	28.74
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp15	30.60	29.67
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp16	19.99	31.23
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp30	37.02	36.86
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp32	28.69	36.60
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp64	32.08	38.95
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp65	38.69	38.91
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp120	38.93	40.36
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp128	31.78	40.19
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp130	38.21	40.24
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp240	38.18	40.76
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp255	38.87	40.66
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp256	36.77	40.43
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp510	37.74	40.09
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp512	36.70	40.08
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp1024	36.44	39.03
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp1025	35.92	38.50
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp2048	35.78	37.95
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	tg128	2.51	2.51

0 replies

Djip007 · 2024-12-08T03:13:08Z

Djip007
Dec 8, 2024
Author

@jart do you want I try it on llamafile ?

1 reply

Djip007 Dec 11, 2024
Author

#655 ...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

optim on tinyblas #654

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

optim on tinyblas #654

Uh oh!

Uh oh!

Djip007 Dec 8, 2024

Replies: 2 comments · 1 reply

Uh oh!

Uh oh!

Djip007 Dec 8, 2024 Author

Uh oh!

Uh oh!

Djip007 Dec 8, 2024 Author

Uh oh!

Djip007 Dec 11, 2024 Author

Djip007
Dec 8, 2024

Replies: 2 comments 1 reply

Djip007
Dec 8, 2024
Author

Djip007
Dec 8, 2024
Author

Djip007 Dec 11, 2024
Author