-
-
Couldn't load subscription status.
- Fork 10.9k
[Bugfix]fix Qwen3 xml tool parser #26345
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bugfix]fix Qwen3 xml tool parser #26345
Conversation
Signed-off-by: Zhikaiiii <1658973216@qq.com>
|
This pull request has merge conflicts that must be resolved before it can be |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request refactors the Qwen3 XML tool parser to fix a bug, likely related to streaming tool calls, by improving its compatibility with serving_chat.py. The changes include switching to a centralized make_tool_call_id utility for consistency and adding state tracking attributes (prev_tool_call_arr, streamed_args_for_tool) that are now correctly managed in both streaming and non-streaming contexts. While the fix for streaming appears correct, I've identified a critical type inconsistency in the non-streaming extract_tool_calls method that could lead to runtime errors.
| self.prev_tool_call_arr.append({ | ||
| "name": "", | ||
| "arguments": {} | ||
| }) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a type inconsistency in how the arguments field is handled for self.prev_tool_call_arr. It is initialized as a dictionary ({}), but later at line 1116, it is assigned a string value from tool_call.function.arguments. This mismatch can lead to TypeError exceptions in downstream code that expects a consistent type. The corresponding logic in the extract_tool_calls_streaming method correctly initializes arguments as an empty string (""). To ensure type consistency and prevent potential runtime errors, you should initialize arguments as an empty string here as well.
| self.prev_tool_call_arr.append({ | |
| "name": "", | |
| "arguments": {} | |
| }) | |
| self.prev_tool_call_arr.append({ | |
| "name": "", | |
| "arguments": "" | |
| }) |
Signed-off-by: Zhikaiiii <1658973216@qq.com>
Signed-off-by: Zhikaiiii <1658973216@qq.com>
| self.streamed_args_for_tool[tool_index] += ( | ||
| tool_call.function.arguments | ||
| ) | ||
| return result |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add corresponding test case to better testing and coverage
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
Signed-off-by: Zhikaiiii <1658973216@qq.com>
Signed-off-by: Zhikaiiii <1658973216@qq.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks
Signed-off-by: Zhikaiiii <1658973216@qq.com> Signed-off-by: bbartels <benjamin@bartels.dev>
Signed-off-by: Zhikaiiii <1658973216@qq.com>
Signed-off-by: Zhikaiiii <1658973216@qq.com>
Signed-off-by: Zhikaiiii <1658973216@qq.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Signed-off-by: Zhikaiiii <1658973216@qq.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Signed-off-by: Zhikaiiii <1658973216@qq.com> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
Signed-off-by: Zhikaiiii <1658973216@qq.com> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
Purpose
Test Plan
We run the unit-test to ensure the parser result is correct.
We also launch an api-server and use a tool-call curl to ensure the problem is fixed.
Test Result
Unit test results:
The respons of api-server
data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"prompt_token_ids":null} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"tool_calls":[{"id":"chatcmpl-tool-d4cfb0be25674db7939e2f08390975c8","type":"function","index":0,"function":{"name":"get_weather","arguments":""}}]},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null,"tool_calls":[{"id":"chatcmpl-tool-d4cfb0be25674db7939e2f08390975c8","type":"function","index":0,"function":{"name":null,"arguments":"{\"location\": \""}}]},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"tool_calls":[{"id":"chatcmpl-tool-d4cfb0be25674db7939e2f08390975c8","type":"function","index":0,"function":{"name":null,"arguments":"北京"}}]},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"tool_calls":[{"id":"chatcmpl-tool-d4cfb0be25674db7939e2f08390975c8","type":"function","index":0,"function":{"name":null,"arguments":""}}]},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"tool_calls":[{"id":"chatcmpl-tool-d4cfb0be25674db7939e2f08390975c8","type":"function","index":0,"function":{"name":null,"arguments":"\""}}]},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":null},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"tool_calls":[{"id":"chatcmpl-tool-d4cfb0be25674db7939e2f08390975c8","type":"function","index":0,"function":{"name":null,"arguments":"}"}}]},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"tool_calls":[{"id":"chatcmpl-tool-d4cfb0be25674db7939e2f08390975c8","type":"function","index":0,"function":{"name":null,"arguments":""}}]},"logprobs":null,"finish_reason":null,"token_ids":null}]} data: {"id":"chatcmpl-9b7984d842c943638ce8675c4688997f","object":"chat.completion.chunk","created":1759831811,"model":"qwen3_coder","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":"tool_calls","stop_reason":null,"token_ids":null}]} data: [DONE]