[Misc] Clean up input processing #17582
Conversation
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337
added
the
ready
DarkLight1337
requested review from
ywang96,
zhuohan123,
youkaichao,
alexm-redhat,
comaniac and
njhill
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
| def _validate_token_prompt(self, prompt: PromptType, | ||
| tokenizer: AnyTokenizer): | ||
| # Guard against out-of-vocab tokens. | ||
| # For some tokenizers, tokenizer.decode will happily return empty text | ||
| # for token ids that are out of vocab, and we don't detect token ids | ||
| # that are greater than the max token id before running the model. | ||
| # However, these token ids will later crash a cuda kernel at runtime | ||
| # with an index out of bounds error. This will crash the entire engine. | ||
| # This needs to happen before multimodal input pre-processing, which | ||
| # may add dummy <image> tokens that aren't part of the tokenizer's | ||
| # vocabulary. | ||
| if is_token_prompt(prompt): | ||
| prompt_ids = prompt["prompt_token_ids"] | ||
| if len(prompt_ids) == 0: | ||
| # Empty prompt check is handled later | ||
| return | ||
| max_input_id = max(prompt_ids) | ||
| if max_input_id > tokenizer.max_token_id: | ||
| raise ValueError( | ||
| "Token id {} is out of vocabulary".format(max_input_id)) | ||
|
|
There was a problem hiding this comment.
#11980 lets us move this to after processor is applied, making V0 validation (_validate_model_inputs) same as V1.
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337
requested review from
WoosukKwon and
robertgshaw2-redhat
as code owners
| cache_salt: NotRequired[str] | ||
| """ | ||
| Optional cache salt to be used for prefix caching. | ||
| """ | ||
|
|
There was a problem hiding this comment.
Although we don't support embedding inputs in V1 yet, I added this so we don't forget to assign cache_salt during processing when we do implement it
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: minpeter <kali2005611@gmail.com>
Following #17045 and #15428, the preprocessing logic has gotten rather complicated. This PR cleans that up by narrowing the prompt types early and handling them separately so that it becomes easier to keep track how each prompt type is being processed .