[v1][KVCacheManager] Avoid full cache hit by controlling max_length #17999
Conversation
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
heheda12345
requested review from
WoosukKwon,
robertgshaw2-redhat,
njhill,
ywang96,
comaniac and
alexm-redhat
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com> Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
|
@heheda12345 Please fix the pre-commit errors. I didn't consider the line length 😅 |
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
…llm-project#17999) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
…llm-project#17999) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>
…llm-project#17999) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> Signed-off-by: minpeter <kali2005611@gmail.com>
When prompt length is divisible by the block size and all blocks are cached, we need to recompute the last token. Instead of handling it by removing the last block hash, it can also be achieved by set max_cache_hit_length to prompt_length - 1 and pass max_cache_hit_length to find_longest_cache_hit.
The API change of find_longest_cache_hit will also help hybrid allocator to get the intersection for prefix cache hit of different layers.