[Bug]: (regression from v0.8.5) missing "finish_reason": null in streaming chat completion outputs

[cjackal]

Your current environment

Hardware / python dependencies are not relevant for this issue.

🐛 Describe the bug

Streaming chat completion outputs in openai-compatible serving since vllm==0.9.0 does not include the "finish_reason": null field in its HTTP response stream like the following:

data: {"id":"chatcmpl-c4eefdc442064e16be757120a2ec7703","object":"chat.completion.chunk","created":1749904375,"model":"meta-llama/llama-4-scout-17b-16e-instruct","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-c4eefdc442064e16be757120a2ec7703","object":"chat.completion.chunk","created":1749904375,"model":"meta-llama/llama-4-scout-17b-16e-instruct","choices":[{"index":0,"delta":{"content":"안","tool_calls":[]}}]}

data: {"id":"chatcmpl-c4eefdc442064e16be757120a2ec7703","object":"chat.completion.chunk","created":1749904375,"model":"meta-llama/llama-4-scout-17b-16e-instruct","choices":[{"index":0,"delta":{"content":"녕","tool_calls":[]}}]}

...

Expected chat completion output (i.e. output from vllm==0.8.5.post1) for comparison:

data: {"id":"chatcmpl-0c0dd0fa4fae4544998222ce6edde854","object":"chat.completion.chunk","created":1749904312,"model":"meta-llama/llama-4-scout-17b-16e-instruct","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-0c0dd0fa4fae4544998222ce6edde854","object":"chat.completion.chunk","created":1749904312,"model":"meta-llama/llama-4-scout-17b-16e-instruct","choices":[{"index":0,"delta":{"content":"안"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-0c0dd0fa4fae4544998222ce6edde854","object":"chat.completion.chunk","created":1749904312,"model":"meta-llama/llama-4-scout-17b-16e-instruct","choices":[{"index":0,"delta":{"content":"녕"},"logprobs":null,"finish_reason":null}]}

...

According to the official OpenAI API reference, finish_reason field should be given for each chat completion chunks; see the streaming response example there. It is also incompatible with the current vLLM's text completion chunks:

data: {"id":"cmpl-fcf5cc0826e8476d962e9c5f4130b368","object":"text_completion","created":1749906268,"model":"meta-llama/llama-4-scout-17b-16e-instruct","choices":[{"index":0,"text":" 난","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-fcf5cc0826e8476d962e9c5f4130b368","object":"text_completion","created":1749906268,"model":"meta-llama/llama-4-scout-17b-16e-instruct","choices":[{"index":0,"text":" 누","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-fcf5cc0826e8476d962e9c5f4130b368","object":"text_completion","created":1749906268,"model":"meta-llama/llama-4-scout-17b-16e-instruct","choices":[{"index":0,"text":"군","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

Text completion chunks preserves "finish_reason":null field as shown above.

This behavioral change was introduced in #17340, where the options for json-dumping is changed from ChatCompletionStreamResponse.model_dump_json(exclude_unset=True) to ChatCompletionStreamingResponse.model_dump_json(exclude_none=True). It seems we need a better way to deal with the DeltaToolCall handling than simple exclude_none.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[cjackal]

@maxdebayser @aarnphm @DarkLight1337 ping to the reviewers of the aforementioned PR. This behavior change conflicts with a few openai-compatible apps (only a few, as most chatbot apps only check whether the finish reason is a nonempty string). I do not use function calling frequently so I just take a simple-minded approach to revert the json dump option to exclude_unset=True for my server, but it may require a proper fix on main branch.

[chaunceyjiang]

I will try to reproduce this issue locally and attempt to fix it.

[chaunceyjiang]

Could you share your client code? I’m unable to reproduce the issue locally. As you can see, each of mine has finish_reason=None.

vllm serve /home/jovyan/qwen3-8b

python examples/online_serving/openai_chat_completion_client.py --stream

ChatCompletionChunk(id='chatcmpl-523a941806c14a4e88ce62491f545553', choices=[Choice(delta=ChoiceDelta(content=' the', function_call=None, refusal=None, role=None, tool_calls=[]), finish_reason=None, index=0, logprobs=None)], created=1749977097, model='/home/jovyan/qwen3-8b', object='chat.completion.chunk', service_tier=None, system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chatcmpl-523a941806c14a4e88ce62491f545553', choices=[Choice(delta=ChoiceDelta(content=' games', function_call=None, refusal=None, role=None, tool_calls=[]), finish_reason=None, index=0, logprobs=None)], created=1749977097, model='/home/jovyan/qwen3-8b', object='chat.completion.chunk', service_tier=None, system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chatcmpl-523a941806c14a4e88ce62491f545553', choices=[Choice(delta=ChoiceDelta(content=' were', function_call=None, refusal=None, role=None, tool_calls=[]), finish_reason=None, index=0, logprobs=None)], created=1749977097, model='/home/jovyan/qwen3-8b', object='chat.completion.chunk', service_tier=None, system_fingerprint=None, usage=None)

[cjackal]

Could you share your client code? I’m unable to reproduce the issue locally. As you can see, each of mine has finish_reason=None.

vllm serve /home/jovyan/qwen3-8b

python examples/online_serving/openai_chat_completion_client.py --stream

ChatCompletionChunk(id='chatcmpl-523a941806c14a4e88ce62491f545553', choices=[Choice(delta=ChoiceDelta(content=' the', function_call=None, refusal=None, role=None, tool_calls=[]), finish_reason=None, index=0, logprobs=None)], created=1749977097, model='/home/jovyan/qwen3-8b', object='chat.completion.chunk', service_tier=None, system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chatcmpl-523a941806c14a4e88ce62491f545553', choices=[Choice(delta=ChoiceDelta(content=' games', function_call=None, refusal=None, role=None, tool_calls=[]), finish_reason=None, index=0, logprobs=None)], created=1749977097, model='/home/jovyan/qwen3-8b', object='chat.completion.chunk', service_tier=None, system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chatcmpl-523a941806c14a4e88ce62491f545553', choices=[Choice(delta=ChoiceDelta(content=' were', function_call=None, refusal=None, role=None, tool_calls=[]), finish_reason=None, index=0, logprobs=None)], created=1749977097, model='/home/jovyan/qwen3-8b', object='chat.completion.chunk', service_tier=None, system_fingerprint=None, usage=None)

The issue does not affect openai python SDK clients, as openai-python automatically sets the default finish_reason to None in its pydantic Choice model definition.

What I have posted above are the raw HTTP response outputs, you may reproduce them with curl -XPOST http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"meta-llama/llama-4-scout-17b-16e-instruct","messages":[{"role":"user","content":"안녕?"}],"stream":true}'

[chaunceyjiang]

Indeed, I can also reproduce it locally. I noticed that other places are using model_dump_json(exclude_unset=True), but this one uses chunk.model_dump_json(exclude_none=True).

I suggest making it consistent with the rest and using model_dump_json(exclude_unset=True) everywhere.

[cjackal]

Indeed, I can also reproduce it locally. I noticed that other places are using model_dump_json(exclude_unset=True), but this one uses chunk.model_dump_json(exclude_none=True).

I suggest making it consistent with the rest and using model_dump_json(exclude_unset=True) everywhere.

Indeed, I consent with what you mentioned on using consistent json dumping behavior overall. But according to #17340 exclude_none=True is there for a reason, so I guess you may need to handle the "designated" tool_choice case; corresponding code block is here

[chaunceyjiang]

@cjackal I did a test, and I believe it's compatible as well. Please take a look at the test results I included in the PR.