Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[KVCache] Initialize one extra page than specified (apache#16849)
This PR udpates PagedKVCache to initialize one more page than specified via constructor. The reason is that applications usually depends the number of free pages (returned from `GetNumAvailablePages`) to decide the KV cache operation policy. If there is no this extra page, the KV cache will tell "no available" pages even when the last allocated pages are not full, which may give the applications an illusion that the KV cache is already completely full, and cause further issues.
- Loading branch information