Windows 64-bit, Microsoft Visual Studio - it works like a charm after those fixes!

[bsiminski]

First of all thremendous work Georgi! I managed to run your project with a small adjustments on:

• Intel(R) Core(TM) i7-10700T CPU @ 2.00GHz / 16GB as x64 bit app, it takes around 5GB of RAM.

Here is the list of those small fixes:

• main.cpp: added ggml_time_init() at start of main (division by zero otherwise)
• quantize.cpp: same as above at start of main (division by zero otherwise)
• ggml.c: #define QK 32 moved to dedicated define.h (should not be in .c)
• ggml.c: replace fopen with fopen_s (VS secure error message)
• ggml.c: below changes due to 'expression must be a pointer or complete object type':

1. 2x (uint8_t*)(y to: ((uint8_t*)y
2. 4x (const uint8_t*)(x to ((const uint8_t*)x
3. 2x (const uint8_t*)(y to ((const uint8_t*)y

• quantize.cpp: removed qk in ggml_quantize_q4_0 & ggml_quantize_q4_1 calls
• utils.cpp: use of QK value instead of parameter value (VS raise error for: uint8_t pp[qk / 2];)

It would be really great if you could incorporate those small fixes.

[etra0]

Interesting, doing these changes (and a couple of more hacks) I was able to run the 13B model on my HW (AMD Ryzen 7 3700X 8-Core Processor, 3593 Mhz, 8 Core(s), 16 Logical Processor(s), 32gb RAM) and I was able to get 268ms per token, with around 8GB of ram usage!

I forced the usage of AVX2 and that gave a huge speed up.

[bsiminski]

@etra0 here are my 13B model tests, based on number of threads & AVX2 (thanks!):

4: 3809.57 ms per token (default settings)
8: 3617.09 ms per token (default settings)
12: 2967.79 ms per token (default settings)

4: 495.08 ms per token (with AVX2)
8: 519.78 ms per token (with AVX2)
12: 490.53 ms per token (with AVX2)

Clearly AVX2 gives a huge boost. I see however that you are still way ahead with your 268 ms. What other optimizations do you have?

[ggerganov]

Yes, AVX2 flags are very important for high performance.
Could you wrap these changes in a PR?

ggml.c: #define QK 32 moved to dedicated define.h (should not be in .c)

This is not very desirable - I don't want an extra file added. Although the QK constants everywhere are indeed problematic.
Some other fix?

[etra0]

Could you wrap these changes in a PR?

I could do that, but I'm unsure whether to create a Solution, or move the project to CMake, because Windows doesn't support Make by default, sadly.

I always try to avoid Solutions because they're not multiplatform, but from looking at the makefile, rewriting it to CMake would take a bit more time. In the meantime I could do a PR to fix the things that won't compile.

[ggerganov]

CMake is better than Solutions. The https://github.com/ggerganov/whisper.cpp project has a CMake build system that is compatible for Windows and the project is very similar. It should be easy to adapt

[kamyker]

Great! These changes finally fixed compilation for me using VS cl command (#2) and also cmake with @etra0 repo.

I get 140ms per token on i9900k and about 5gb ram usage with 7B.

Unfortunately bigger prompts are kind of unusable. Dno if it's windows issue or this library isn't yet optimized in this case. Making hardcoded 512 token limit a parameter was easy to change but it's too slow as it repeats all prompt tokens.

[ggerganov]

@kamyker

Maybe the context size has to be increased - it's currently hardcoded to 512:

https://github.com/ggerganov/llama.cpp/blob/da1a4ff01f42d058cfa59806dd5679c0fe5a8604/main.cpp#L768

Haven't tested if it works with other values