scheduler max-length

[jth3galv]

I am running mpt-7b-instruct which should support "arbitrary long contexts". However when I provide the model long inputs I get this warning (and the model does not produce any output):

Token indices sequence length is longer than the specified maximum sequence length for this model (3622 > 2048). Running this sequence through the model will result in indexing errors
WARNING 07-12 15:03:35 scheduler.py:194] Input prompt (3622 tokens) is too long and exceeds limit of 2560

What is this scheduler limit? My understanding is that it is not related to the model. How can I change it?

[LiuXiaoxuanPKU]

Hi, the maximum sequence length is decided by the maximum sequence length allowed by the model and the maximum number of batched tokens as shown here.

It seems you hit the limit of the maximum number of batched tokens. You could change the default value here.

[jth3galv]

Thank you so much for the super fast reply.

That worked. Although I now have:

Token indices sequence length is longer than the specified maximum sequence length for this model (3651 > 2048). Running this sequence through the model will result in indexing errors

In configs.json I have:

"max_seq_len": 8192,

Am I doing something wrong?

[jth3galv]

@LiuXiaoxuanPKU I've tracked down the issue. It seems that the tokenizer is not passed the correct model_max_length value.

In llm_engine.py at line 81 we have:

    self.tokenizer = get_tokenizer(
        model_config.tokenizer,
        tokenizer_mode=model_config.tokenizer_mode,
        trust_remote_code=model_config.trust_remote_code)

adding model_max_length= "value" did the trick.

I think that should be read from the corresponding config.json attribute.

[LiuXiaoxuanPKU]

Sorry for the late reply. Yeah, we did not pass the model_max_length directly into the tokenizer because we want to reject this request. Instead of letting the tokenizer truncate the prompt and accept the request, the engine (more precisely, scheduler) will recognize that this request has a long input prompt and ignore the request. Please let me know if you find more problems.

[jth3galv]

Thanks for your reply!

A few comments:

Would it not be better for the tokenizer to be aligned with the model?

That is: if I set the max_seq_len in the model config to some value I think it would be nice to have the tokenizer max length set to the same value.

If this is not the desirable behavior, would it be possible to set the tokenizer max_seq_len somewhere else without having to modify the code?

[jmzeng]

What would be the recommended max_num_batched_tokens for 8 H100s with 80GB GPU each?

[AmoghM]

Running into the same problem while running llama-2.

For input prompt token length of 3503 and setting max_tokens to 2000, I got the warning:
“Input prompt (3503 tokens) is too long and exceeds limit of 2560”.

We would get that message when num_prompt_token > prompt_limit where prompt_limit is min (max_model_len and max_num_batched_tokens). By default, the max_num_batched_tokens is set to 2560.

So I increased the max_num_batched_tokens to 5000 to see what happens next. This time I get Token indices sequence length is longer than the specified maximum sequence length for this model (3503 > 2048). Running this sequence through the model will result in indexing errors

My questions are:

• Why did it pick up 2048 and not 4096 for llama-2 as that is its max context window?
• Why we don't send a 500 error code but 200? The server replies back the same input prompt/context as output with a 200 OK status.

[rndlr96]

Did you set bot max_model_len and max_num_batched_tokens?

FYI.

vllm/vllm/core/scheduler.py

Line 66 in 6428f1d

self.prompt_limit = min(self.scheduler_config.max_model_len,

[diego898]

@LiuXiaoxuanPKU

Sorry for the late reply. Yeah, we did not pass the model_max_length directly into the tokenizer because we want to reject this request. Instead of letting the tokenizer truncate the prompt and accept the request, the engine (more precisely, scheduler) will recognize that this request has a long input prompt and ignore the request. Please let me know if you find more problems.

Hello! So to confirm/ask a question: vllm rejecting any inputs longer than the specified max-model-len parameter is a design decision, and there is no plan to support truncation (on left or right side) for inputs longer than this length?

[njhill]

There is an open PR to add an input truncation option: #3144