Skip to content

Introduce Q8_0 and Q4_0 with Bf16 delta values #7497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Introduce Q8_0 and Q4_0 with Bf16 delta values #7497

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
e26fd70
Introduce Q4_0 and Q8_0 quantizations with BF16 delta values
Srihari-mcw Aug 12, 2024
983b03a
Add additional comments
Srihari-mcw May 23, 2024
9e5174c
Remove additional ifdef conditions
Srihari-mcw May 24, 2024
c480818
Fix issues with SSE3 version for vec_dot_q4_0_b16_q8_0_b16
Srihari-mcw May 24, 2024
5a6a235
Fix build issues in sgemm.cpp post rebase
Srihari-mcw Jul 8, 2024
db6657e
Fic more conflicts in quantize.cpp
Srihari-mcw Jul 30, 2024
f7ce132
Add changes to fix compiler issues
Srihari-mcw Jul 15, 2024
ef693e9
Add the new data types across files
Srihari-mcw Aug 12, 2024
4a2f703
Add changes to use union data type for better conversion to strong ty…
Srihari-mcw Aug 2, 2024
7927655
Update the type name in llama.cpp
Srihari-mcw Aug 6, 2024
43c7be5
Add the BF16 delta data types in constants.py
Srihari-mcw Aug 12, 2024
806c5a4
Remove additional snippets for CI/CD issues with python constants.py …
Srihari-mcw Aug 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions examples/quantize/quantize.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ struct quant_option {

static const std::vector<struct quant_option> QUANT_OPTIONS = {
{ "Q4_0", LLAMA_FTYPE_MOSTLY_Q4_0, " 4.34G, +0.4685 ppl @ Llama-3-8B", },
{ "Q4_0_B16", LLAMA_FTYPE_MOSTLY_Q4_0_B16, " 3.56G, 5.9624 +/- 0.03348 ppl @ LLaMA-v2-7B", },
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The perplexity score mentioned here was derived from running perplexity.exe with Q4_0_B16 quantized model for meta llama2 7B model. Was unsure of methodology for the other ppl scores put across here. Kindly share feedback if the score put here needs to be modified. Thanks

{ "Q4_1", LLAMA_FTYPE_MOSTLY_Q4_1, " 4.78G, +0.4511 ppl @ Llama-3-8B", },
{ "Q5_0", LLAMA_FTYPE_MOSTLY_Q5_0, " 5.21G, +0.1316 ppl @ Llama-3-8B", },
{ "Q5_1", LLAMA_FTYPE_MOSTLY_Q5_1, " 5.65G, +0.1062 ppl @ Llama-3-8B", },
Expand Down Expand Up @@ -46,6 +47,7 @@ static const std::vector<struct quant_option> QUANT_OPTIONS = {
{ "Q5_K_M", LLAMA_FTYPE_MOSTLY_Q5_K_M, " 5.33G, +0.0569 ppl @ Llama-3-8B", },
{ "Q6_K", LLAMA_FTYPE_MOSTLY_Q6_K, " 6.14G, +0.0217 ppl @ Llama-3-8B", },
{ "Q8_0", LLAMA_FTYPE_MOSTLY_Q8_0, " 7.96G, +0.0026 ppl @ Llama-3-8B", },
{ "Q8_0_B16", LLAMA_FTYPE_MOSTLY_Q8_0_B16, " 6.70G, 5.8011 +/- 0.03239 ppl @ LLaMA-v1-7B", },
{ "Q4_0_4_4", LLAMA_FTYPE_MOSTLY_Q4_0_4_4, " 4.34G, +0.4685 ppl @ Llama-3-8B", },
{ "Q4_0_4_8", LLAMA_FTYPE_MOSTLY_Q4_0_4_8, " 4.34G, +0.4685 ppl @ Llama-3-8B", },
{ "Q4_0_8_8", LLAMA_FTYPE_MOSTLY_Q4_0_8_8, " 4.34G, +0.4685 ppl @ Llama-3-8B", },
Expand Down
4 changes: 4 additions & 0 deletions ggml/include/ggml.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -391,6 +391,8 @@ extern "C" {
GGML_TYPE_Q4_0_4_4 = 31,
GGML_TYPE_Q4_0_4_8 = 32,
GGML_TYPE_Q4_0_8_8 = 33,
GGML_TYPE_Q4_0_B16 = 34,
GGML_TYPE_Q8_0_B16 = 35,
GGML_TYPE_COUNT,
};

Expand Down Expand Up @@ -435,6 +437,8 @@ extern "C" {
GGML_FTYPE_MOSTLY_Q4_0_4_4 = 25, // except 1d tensors
GGML_FTYPE_MOSTLY_Q4_0_4_8 = 26, // except 1d tensors
GGML_FTYPE_MOSTLY_Q4_0_8_8 = 27, // except 1d tensors
GGML_FTYPE_MOSTLY_Q4_0_B16 = 28, // except 1d tensors
GGML_FTYPE_MOSTLY_Q8_0_B16 = 29, // except 1d tensors
};

// available tensor operations:
Expand Down
11 changes: 11 additions & 0 deletions ggml/src/ggml-impl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,13 @@
#if defined(_MSC_VER)

#define m512bh(p) p
#define m128bh(p) p
#define m512i(p) p

#else

#define m512bh(p) (__m512bh)(p)
#define m128bh(p) (__m128bh)(p)
#define m512i(p) (__m512i)(p)

#endif
Expand Down Expand Up @@ -100,6 +102,15 @@ static inline ggml_bf16_t ggml_compute_fp32_to_bf16(float s) {
return h;
}

static inline ggml_bf16_t ggml_make_bf16(uint16_t h) {
union {
ggml_bf16_t f;
uint16_t i;
} u;
u.i = h;
return u.f;
}

#define GGML_FP32_TO_BF16(x) ggml_compute_fp32_to_bf16(x)
#define GGML_BF16_TO_FP32(x) ggml_compute_bf16_to_fp32(x)

Expand Down
Loading
Loading