[MISC] Add prefix cache reset to LMCache CPU offload example #27248
Conversation
|
Documentation preview: https://vllm--27248.org.readthedocs.build/en/27248/ |
mergify
bot
added
documentation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request correctly adds a call to reset_prefix_cache in the LMCache CPU offload example to ensure both save and load operations are demonstrated. This is a good improvement for the example's clarity. I've added one suggestion to make the example more robust by checking the return value of the cache reset operation.
| time.sleep(1) | ||
| # Clear vLLM's internal prefix cache to force the second request | ||
| # to fetch cached KVs from LMCache | ||
| llm.reset_prefix_cache() |
There was a problem hiding this comment.
The reset_prefix_cache method can fail if there are still blocks in use, and it returns a boolean indicating success or failure. The example should check this return value and raise an error if it's False. This will make the example more robust and prevent confusion if the cache reset fails, which would cause the example to not demonstrate the intended LMCache loading behavior.
| llm.reset_prefix_cache() | |
| if not llm.reset_prefix_cache(): | |
| raise RuntimeError( | |
| "Failed to reset prefix cache. The example may not run as expected." | |
| ) |
Signed-off-by: zhou.qianjun <zhou.qianjun@zte.com.cn>
1 participant
Purpose
To trigger both LMCache save and load operations in this example.
Test Plan
Test Result
before
The second request only triggers a cache hit in LMCache, still using the local cache.
after
After clearing the prefix cache from the first request, the logs will show that the second request hits and loads the KV cache from LMCache.
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.