Server hanging after prompt exceeds limit with LLaMA 2 models

[dexterju27]

Hello,

I have been trying vllm 0.1.3 with LLaMA 2 model,

python  -m vllm.entrypoints.openai.api_server  --model meta-llama/Llama-2-70b-chat-hf --port 6011 --tensor-parallel-size 8 --tokenizer hf-internal-testing/llama-tokenizer

I noticed the server will hang and stop processing further requests after a prompt being too long. The last output is:

WARNING 08-15 11:44:47 scheduler.py:130] Input prompt (2715 tokens) is too long and exceeds limit of 2560

I see there is a related fix: #273 but not sure why this still happens.

Is there a way to change this behavior and just discard this request instead?

Thanks!

[amoghmishra-sl]

Facing the exact same thing @dexterju27!

Running into the same problem while running llama-2.

For input prompt token length of 3503 and setting max_tokens to 2000, I got the warning:
“Input prompt (3503 tokens) is too long and exceeds limit of 2560”.

We would get that message when num_prompt_token > prompt_limit where prompt_limit is min (max_model_len and max_num_batched_tokens). By default, the max_num_batched_tokens is set to 2560.

So I increased the max_num_batched_tokens to 5000 to see what happens next. This time I get Token indices sequence length is longer than the specified maximum sequence length for this model (3503 > 2048). Running this sequence through the model will result in indexing errors

My questions are:

Why did it pick up 2048 and not 4096 for llama-2 as that is its max context window?
Why we don't send a 500 error code but 200? The server replies back the same input prompt/context as output with a 200 OK status.

[dalong2hongmei]

like this issue. i ran baichuan13b using vllm.entrypoints.api_server, it hang and all gpus utilization became zero.

[amoghmishra-sl]

@dalong2hongmei can you share your logs? I have seen mine used to get stuck after it went beyond 94% of cache utilization.

[horiacristescu]

Got the same error "exceeds limit of 2560"

[efwfe]

Got the same error

[GasolSun36]

anyone has solutions？I get the same error.

[esmeetu]

set args --max-num-batched-tokens larger when run server, like 8192.

if not, try buld latest main branch.

[xingyaoww]

I was having this issue with vllm==0.1.3, upgrading to vllm==0.1.4 fixes this issue for me.

[hmellor]

Closing as this appears to be resolved.