Skip to content

Commit 85c5280

Browse files
committed
llamafile_sgemm API - INT8 implementation
This change upstreams llamafile's cpu matrix multiplication kernels for ppc64le using MMA builtins for quantised int8 datatype. This change results in 10% - 70% improvement in total speed(ie all tokens/total time), across various batch sizes. The patch is tested with Meta-Lllama-3-8B, Mistral-7B, Llama-2-7B-chat-hf models on a IBM POWER10 machine. Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
1 parent d408bb9 commit 85c5280

File tree

2 files changed

+649
-69
lines changed

2 files changed

+649
-69
lines changed

0 commit comments

Comments
 (0)