[Bugfix][V0] Multi-sequence logprobs streaming edge case #15259
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
andylolu2
changed the title
andylolu2
changed the title
There was a problem hiding this comment.
Thanks for the fix @andylolu2!
BTW this is on the v0 path which we're aiming to deprecate soon anyhow. Hopefully it's not an issue in v1.
njhill
changed the title
Signed-off-by: Andy Lo <andy@mistral.ai>
Signed-off-by: Andy Lo <andy@mistral.ai>
andylolu2
force-pushed
the
andy/fix-multiseq-logprobs
branch
from
80327d1 to
763405f
Compare
|
Thanks @njhill! I've cleaned up the PR |
…t#15259) Signed-off-by: Andy Lo <andy@mistral.ai>
…t#15259) Signed-off-by: Andy Lo <andy@mistral.ai>
…t#15259) Signed-off-by: Andy Lo <andy@mistral.ai> Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>
…t#15259) Signed-off-by: Andy Lo <andy@mistral.ai>
…t#15259) Signed-off-by: Andy Lo <andy@mistral.ai>
…t#15259) Signed-off-by: Andy Lo <andy@mistral.ai>
…t#15259) Signed-off-by: Andy Lo <andy@mistral.ai> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
There's a bug when using logprobs with multi-sequence with RequestOutputKind.DELTA. When a sequence finishes before the others, it's
output_token_idson the subsequent steps will be an empty array, until every sequence finishes. This cases an[-0:]slice into the logprobs array, which returns the entire logprobs array instead of the 0-length suffix.Adding an if check to handle this edge case.