paged attention in tgi is fake?

[CSEEduanyu]

System Info

when max_new_token is very big,kv cache can be seriously wasteful

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

max_new_token=8192

Expected behavior

max_new_token does not affect the actual memory utilization

[CSEEduanyu]

pagedattention is used in tgi but no supporting work is done, so kv cache memory will be seriously wasted

[CSEEduanyu]

Are there plans to change this? Or is there any information I can refer to if I transform first

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.