Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Xeon Phi (Knights Corner) Support. #6440

Open
wants to merge 213 commits into
base: master
Choose a base branch
from
Open

Xeon Phi (Knights Corner) Support. #6440

wants to merge 213 commits into from

Conversation

julialongtin
Copy link

@julialongtin julialongtin commented Apr 2, 2024 •
edited
Loading

Most of the gains come from an assembly implementation of Q5K . Q8K dot product code, written in IMCI assembly.

goes from 0.18 tokens per second on mistral 7B instruct (Q5K) to 1.2 tokens per second.

@julialongtin
add detection of Xeon PHI: Knights Corner.
868a201
@julialongtin
handle the case that we have no glibc on the PHI.
7f3722b
@julialongtin
instead of checking on glibc, check on SYS_getcpu
5a2973a
@julialongtin
try to detect the PHI cross compiler in make.
a31c936
@julialongtin
try to detect the PHI cross compiler in make.
aec982e
@julialongtin
try to implement one intrinsic
f346a41
@julialongtin
use right type, and define GGML_F32_VEC_ZERO.
a1ae649
@julialongtin
import intrinsics.
7a57feb
@julialongtin
implement F32 dot products.
717e164
@julialongtin
Update ggml.c
257ffd9
@julialongtin
Update ggml.c
e216a2f
@julialongtin
Update ggml.c
eac00a7
@julialongtin
merge from upstream
fe663c1
@julialongtin
add a benchmark / test binary.
f882673
@julialongtin
Update ggml-phi-knc.c
ab6f3a8
@julialongtin
remove intrinsics import, and use upConv to save 12 bytes of memory t…
ee27148 
…ransit.
@julialongtin
use the same header as ggml.c, and remove some warnings.
76e66e7
@julialongtin
formatting changes.
ac36371
@julialongtin
spacing changes.
0979522
@julialongtin
be more specific about the length of our list of run amounts.
9185e14
@julialongtin
begin work on targeting dot_q5_K_q8_K.
a7bd64c
@julialongtin
import stdint.h for sizeSt.
9bcb835
@julialongtin
import stdio.h for size_t.
8f57803
@julialongtin
pull in ggml specific types.
cd20404
@julialongtin
tell ggml-common.h to export what we want.
18f3539
@julialongtin
force to compile.
0b3f171
@julialongtin
allow using code from ggml-phi-knc-dot_q5_K_q8_K.c
0b012c0
@julialongtin
attempt to speed up float clearing.
0a2051a
@julialongtin
first fixes.
6face8a
@julialongtin
formatting improvement.
edb76ff
@julialongtin
perform better prefetches, and invert the test of our clear flag for …
d8d574c 
…clarity.
@julialongtin
remove useless prefetches.
204bc1f
@julialongtin
spacing and comment changes.
f555f9d
@julialongtin
move sub earlier, and move the compare of iterations to outside, and …
dda250f 
…at the end of the loop.
@julialongtin
fix loop.
270204e
@julialongtin
use values inside of the loop as soon as we have them.
9a1a53b
@julialongtin
correct a comment, and use jz when comparing to zero.
f3b86eb
@julialongtin
comment clarification.
4097cde
@julialongtin
change from handling three iterations per loop to four.
511ad80
@julialongtin
subtract the correct amount.
47ca67a
@julialongtin
look at the right final memory location.
1b7ca0b
@julialongtin
add missing jump.
4d94831
@julialongtin
spacing changes.
fc23c22
@julialongtin
spacing changes.
a273a9e
@julialongtin
introduce r10 and r11, for vloadunpackhd.
9f3623f
@julialongtin
rename label 1 to 3.
9aa34c8
@julialongtin
rename some labels.
eefa650
@julialongtin
relabel some other labels.
0c0137e
@julialongtin
fill and increment r12 and r13.
50887fc
@julialongtin
add missing vector.
257c06b
@julialongtin
make the offset of q4 available.
3d39d61
@julialongtin
minor comment fixes.
420e9db
@julialongtin
load from identical addresses for low and high side.
084e368
@julialongtin
make offset available in a register.
7925fb1
@julialongtin
do 2 rounds of 4, instead of 4 rounds of 2. and properly offset unall…
bd22e9d 
…igned reads across a 64 byte boundary.
@julialongtin
spacing changes.
aede2f5
@julialongtin
Merge branch 'master' into 0.99-rebase
ded062c
@julialongtin
Merge pull request #4 from julialongtin/0.99-rebase
03a8b80 
0.99 rebase
@github-actions github-actions bot added the ggml changes relating to the ggml tensor library for machine learning label Jun 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cebtenzzre cebtenzzre cebtenzzre left review comments

@ggerganov ggerganov ggerganov left review comments

@atelepov atelepov atelepov approved these changes

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
enhancement New feature or request ggml changes relating to the ggml tensor library for machine learning performance Speed related topics Review Complexity : High Generally require indepth knowledge of LLMs or GPUs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@julialongtin @Titaniumtown @BarfingLemurs @AutonomicPerfectionist @ggerganov @atelepov @cebtenzzre @mofosyne