ggml-alloc : potential regression with n_batch == 1

[ggerganov]

The following command fails:

./bin/perplexity -m ../models/llama-7b-v2/ggml-model-q4_0.gguf -f ../build/wikitext-2-raw/wiki.test.raw -ngl 0 -t 4 -b 1

llama_new_context_with_model: kv self size  =  256.00 MB
llama_new_context_with_model: compute buffer total size =    1.61 MB
ggml_allocr_alloc: not enough space in the buffer (needed 88064, largest block available 32)
GGML_ASSERT: /Users/ggerganov/development/github/llama.cpp/ggml-alloc.c:174: !"not enough space in the buffer"
Abort trap: 6

I think this used to work until recently.

[ggerganov]

This commit seems to break it 9217721

[ggerganov]

I recently changed the "warm up" run to use 2 tokens instead of 1 because I thought this will "exercise" the non mat-vec kernels and appeared to have some positive effect on timings:

https://github.com/ggerganov/llama.cpp/blob/fec2fb19e4229aac58c98171c46e77144b99f8a3/common/common.cpp#L775

Changing this back to 1 token fixes the crash.

[slaren]

I think this is expected, evaluating a batch larger than the batch size will result in out of memory errors. We really need to add checks for these things in llama_eval. I guess the fix would be to limit the warmup run to n_batch tokens at most.

[ggerganov]

Yup we need some asserts to remind us when we pass inconsistent parameters.
Will fix this with limiting the warm-up run to n_batch and let's add some checks later

[slaren]

Additionally, I think we need to consider that the warmup run will interfere with code that uses llama_get_kv_cache_token_count as the value of n_past. In our examples, simple and beam-search seem to use it. So maybe we need a function to reset kv_self.n to zero.

[slaren]

llama-bench currently only uses a single token for the warmup run. Should it be updated to use 2 tokens when possible?

[ggerganov]

Yup, let's update it. I think it had a positive effect on the time measurements when using Metal