Skip to content

Commit a010502

Browse files
andylolu2erictang000
authored andcommitted
[Bugfix][V0] Multi-sequence logprobs streaming edge case (vllm-project#15259)
Signed-off-by: Andy Lo <andy@mistral.ai>
1 parent af6b7cb commit a010502

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

vllm/outputs.py

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -223,7 +223,12 @@ def from_seq_group(
223223
if delta:
224224
# Slice logprobs delta if applicable
225225
if output_logprobs:
226-
output_logprobs = output_logprobs[-num_output_tokens:]
226+
# num_output_tokens can be 0 when n > 1 and request finishes
227+
# before the others
228+
if num_output_tokens > 0:
229+
output_logprobs = output_logprobs[-num_output_tokens:]
230+
else:
231+
output_logprobs = None
227232
# Don't include prompt if this is after the first output
228233
# containing decode token ids
229234
if include_prompt and seq.get_output_len() > num_output_tokens:

0 commit comments

Comments
 (0)