Please upgrade the KV cache size yes using `--ctx-size`

[enn-nafnlaus]

          Please upgrade the KV cache size yes using `--ctx-size`

Originally posted by in #6603 (comment)

This is not an appropriate response to people having this problem. --ctx-size is a memory-limited operation; of course we'd set it higher if we could. Mine is at 16k and I still hit this problem.

The appropriate response to running out of tokens is to fail the query. It's not for the server to go into an infinite loop and stop all further processing. I've lost days worth of processing time to this bug, when I log into my server and discover that it's no longer running because of this.

The server should never go into an infinite loop; I mean, obviously? If it can't handle a query, it should just reject it and move on.

EDIT: The folk was just running a very outdated server version. Always use --n-predict N to avoid infinite loop.

[phymbert]

Do you have an appropriate response then ?

[phymbert]

You are welcome to submit a new scenario using the server test framework to reproduce any issue you identified. It will help the maintainers understand it and eventually submit a patch.

Contributions are welcomed, and helping the community requires a non null effort.

Regarding the closed discussion, some questions may require more attention than others. I did not notice here that the server was infinite looping because the KV cache limit is reached. I am not sure this is unexpected.

@ggerganov, probably I missed something

[enn-nafnlaus]

Do you have an appropriate response then ?

As stated: " If it can't handle a query, it should just reject it and move on." It shouldn't go into an infinite loop and stop serving requests.

As for reproducing it: I don't know which of my queries are triggering it, as they're randomly generated (within given frameworks) and it queues up many queries in advance (perhaps others might have a specific one?). But you do have the error message. The error is in examples/server/server.cpp, in update_slots(), in response to a failure in llama_decode. The symptoms are that the GPU activity drops to zero and instead the server just rapidly repeats that message while cycling through batch sizes.

I don't think saying "helping the community requires a non null effort" to someone who has never refused a request to help is productive - and nor am I the only person who has brought up this problem here.

My commandline:

. venv/bin/activate
CUDA_VISIBLE_DEVICES=1 ./server --model /path/to/TheBloke_Mixtral-8x7B-Instruct-v0.1-GGUF/mixtral-8x7b-instruct-v0.1.Q3_K_M.gguf --port 1234 --batch-size 2048 --threads 4 --threads-batch 1 --numa --mlock --ctx-size 16384 -cb

So yes, if there's some more datagathering you want from me, given the above information, please let me know how you would like me to go about doing so.

[phymbert]

So yes, if there's some more datagathering you want from me, given the above information, please let me know how you would like me to go about doing so.

You need to provide detailed steps to reproduce your issue. For the server, there is a dedicated framework for that.

Again, if your kv cache is full , it is just not adapted for your usage. You did not provide information to prove that the server goes to an infinite loop in this situation.

If you see failed to find free space in the KV cache, you can try to enable defrag. --defrag-thold 0.1.

I advise to change the way you are requesting for help, we are a team of volunteers here, feedback like that is not appreciable and especially it does not help at all.

[enn-nafnlaus]

All anyone here cares about is the server not going into an infinite loop, which should be considered a de minimis requirement for a server. Closing fixing an infinite loop bug on everyone who brings it up (aka, not just me) as "not planned" is beyond bizarre for a server maintainer. Especially when the server runs a non-deterministic process of which the user has little clue of what the output will be in advance (it's not like users ask the server "please write 20 million tokens", it's going to be "do some simple task" but then the server gets stuck in a loop repeating itself).

This (server #2 in this case) is not okay for a server.

Again: I have never refused a request to gather more data. All I ask is that you not close this bug that's hitting multiple people and paralyzing their servers is "not planned". In the meantime, I'll keep working on seeing if I can figure out a consistently repeatable way to reproduce it. It happens on average once every 15 hours or so to me.

[ggerganov]

@phymbert #6603 was closed reasonably - the user asked how to increase context size and an answer was provided. Similar topic was discussed in #5737 (comment)

@enn-nafnlaus First try to update to latest master. If the problem still persists, you can try to set an upper limit for your queries using the --n-predict argument. By default it is set to -1 which means infinite generation (using context shift) unless EOS token is encountered. My guess is that there are some degenerate cases in which EOS is never emitted which causes the issue that is observed, but without a way to reproduce it's hard to figure out what to do.

[enn-nafnlaus]

@phymbert #6603 was closed reasonably - the user asked how to increase context size and an answer was provided. Similar topic was discussed in #5737 (comment)

@enn-nafnlaus First try to update to latest master. If the problem still persists, you can try to set an upper limit for your queries using the --n-predict argument. By default it is set to -1 which means infinite generation (using context shift) unless EOS token is encountered. My guess is that there are some degenerate cases in which EOS is never emitted which causes the issue that is observed, but without a way to reproduce it's hard to figure out what to do.

Thanks for this :) Can't update to the latest update at the moment (I've moved and the server has no net access), but will try --n_predict to see if that works as a workaround (note: I don't know the number of input tokens in advance on the server, so I'll have to set --n_predict to the max context). And will keep trying to see if I can figure out how to reproduce it. Is there any way to have the server print out what queries it's processing, so I can know what one(s) is/are causing the problem? I see it seems to be creating a file called "llama.log", but I can't specify the filename, so I assume that all running servers write to the same logfile...

[enn-nafnlaus]

You did not provide information to prove that the server goes to an infinite loop in this situation.

I didn't realize that bug reports were about having to "prove" that you're actually experiencing the bug. :Þ

See the size of llama.log.bak (moved from llama.log before I restarted the server) above:

That's from the spam from when it goes into an infinite loop.

Obviously I can't attach a 73 gig file. :) Here's a version truncated after it goes into an infinite loop:

llama.log.bz2.txt

(Remove the .txt extension before uploading)

[phymbert]

@ggerganov, thanks. We have a test feature wrong_usage for this use case where the user forgot to pass n_predict. But I am wondering if it should not be defaulted to n_ctx. Thoughts ?

[slaren]

Are you sure that this is wrong usage? I think the logs hint very strongly that there is an infinite loop in update_slots.

In any case, it is also not ok to generate indefinitely with repeated context shifts if the model never generates an eos. At the very least, that's a denial of service vulnerability.

[phymbert]

Yes, either n_ctx is too small (n_batch=512), either the model hallucinates indefinitely without n_predict set.

[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 256
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 128
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 64
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 32
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 16
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 8
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 4
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 2
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 1
[1712885022] update_slots : failed to decode the batch, n_batch = 1, ret = 1

It is not an infinite loop inside update_slots as slots are beeing released when n_batch == 1.

https://github.com/ggerganov/llama.cpp/blob/91c736015b66ba1d0b82cbae6313b6d5eaa61b68/examples/server/server.cpp#L2187-L2193

[slaren]

Wouldn't you see log messages such as this if it is was truly the model generating indefinitely with context shifts?

{"tid":"140358648172544","timestamp":1712918141,"level":"INFO","function":"update_slots","line":1811,"msg":"slot context shift","id_slot":0,"id_task":404,"n_keep":29,"n_left":2,"n_discard":1,"n_ctx":32,"n_past":31,"n_system_tokens":0,"n_cache_tokens":31}

If the slot was released properly, that should be the end of it, but the same thing happens again and again.

[enn-nafnlaus]

Yes, either n_ctx is too small (512), either the model hallucinates indefinitely without n_predict set.

[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 256
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 128
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 64
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 32
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 16
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 8
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 4
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 2
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 1
[1712885022] update_slots : failed to decode the batch, n_batch = 1, ret = 1

It is not an infinite loop inside update_slots as slots are beeing released when n_batch == 1.

https://github.com/ggerganov/llama.cpp/blob/91c736015b66ba1d0b82cbae6313b6d5eaa61b68/examples/server/server.cpp#L2187-L2193

n_ctx is 16k - see my commandline above.

It very much is an infinite loop if it gets stuck on processing a single query indefinitely, all incoming queries just sit there and the GPU usage drops to 0 while the server spits out spam until I restart it.

The instant it hits n_batch == 1 it jumps back to n_batch 512 failures, continuing over and over in a loop. Spam is pumped out at an immense speed (note that the logfile hit 73 gigs overnight), cycling through this over and over, until I notice that it's gotten into this state and kill the server

[phymbert]

Wouldn't you see log messages such as this if it is was truly the model generating indefinitely with context shifts?

These logs unfortunately are only in stdout not in llama.log.