[Bug]: 400 response when using Qwen3 + reasoning + tool calling + tool_choice "required"

[andysalerno]

Your current environment

docker container tagged image: vllm/vllm-openai:v0.9.0.1

🐛 Describe the bug

I am launching vllm via docker with these args:

      --model Qwen/Qwen3-30B-A3B-FP8
      --reasoning-parser qwen3
      --max-num-seqs 4
      --tool-call-parser hermes
      --enable-auto-tool-choice
      --enable-reasoning
      --served-model-name model
      --max-model-len 32648
      -tp 2

Everything works great - reasoning is parsed, tool calls work (in most scenarios), etc.

But there's one specific scenario where it consistently fails, and the service returns a 400:

when I make a request with "tool_choice": "required", it seems to conflict with the reasoning parsing. I get an error response like so

{'object': 'error', 'message': '1 validation error for list[function-wrap[__log_extra_fields__()]]\n  Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value=\'<think>\\nOkay, the user ...ng set 2025"\\n  }\\n}\\n]\', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid', 'type': 'BadRequestError', 'param': None, 'code': 400}

I guess the relevant part from the above is:

Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value=\'<think>\\nOkay, the user ...ng set 2025"\\n

Note: that above output is technically from the pydantic-ai client, which I am using to make requests to vllm. But, I think the client can't be the cause, because vllm itself is responding with a 400 to my properly-formed request.

I suppose the model is outputting its reasoning in tags as expected, but then the "required" tool choice parsing (guidance or xgrammar or whatever is used) is failing to parse the tool call out from the text, since it includings the reasoning section? Not sure why this would only happen when "tool_choice": "required" but I suppose it has to do with how that interacts with the grammar parsing library.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[chaunceyjiang]

Can you try adding an extra parameter? --guided-decoding-backend xgrammar

[andysalerno]

Can you try adding an extra parameter? --guided-decoding-backend xgrammar

I tried this just now with the same outcome. Here's the relevant logs from vllm btw

[llmhost] | INFO 06-02 21:34:55 [logger.py:42] Received request chatcmpl-9060c008664b44d2843e47ac15fe420d: prompt: '<|im_start|>system\nYou are a research assistant. Perform research to accomplish the given task.\n\nResearch is performed by querying the web and visiting relevant web pages.\n\n## Additional context\nThe current date is: 2025-06-02.\n\n## Rules\n- You MAY invoke multiple tools to gather information related to your task.\n- However, you are limited to at most **5** total tool invocations during this task (since the last user message).\n- After invoking at most **5** tools, you must then invoke the `research_complete` tool to indicate that you are done.\n- Additionally, you must NOT invoke the `research_complete` tool in the same response as other tools. It must be invoked alone.\n- If you see interesting urls in search results, use the `visit_url` tool to scrape them and acquire their information.\n- You MAY invoke any tool multiple times consecutively.\n- If you invoke a tool but it does not provide the information you need, you MAY invoke the same tool again with a different query.\n\n## Definition of done\nYour research is complete when you have gathered sufficient information to respond to the task.\nAt that point, invoke the tool `research_complete` to indicate that you are done. Include a handoff message briefly summarizing what you found.\nIt is not your responsibility to write a summary or report - the user will see all the documents you found.\nSimply invoke `research_complete` to share your findings.\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{"type": "function", "function": {"name": "search_web", "description": "\\nSearch Google for the given query. Returns the search results (including snippet and url) in JSON format.\\nUseful for looking up: current events, up-to-date information, and general knowledge across the web.\\n\\nArgs:\\n    query: The query to search for. Remember, this is a google query, so write your query as you would in Google.\\n", "parameters": {"properties": {"query": {"type": "string"}}, "required": ["query"], "type": "object"}}}\n{"type": "function", "function": {"name": "visit_url", "description": "\\nVisits (scrapes) the given URL and returns the text content of the page.\\nUseful for looking up current events, or visiting urls you found in your research.\\n\\nArgs:\\n    url: The url to visit (scrape).\\n", "parameters": {"properties": {"url": {"type": "string"}}, "required": ["url"], "type": "object"}}}\n{"type": "function", "function": {"name": "execute_python_code", "description": "\\nExecutes Python code and returns the output from stdout.\\nMultiple lines are allowed, including complex scripting.\\nIMPORTANT: you ONLY see the results from stdout, so you MUST print() the result you want to see.\\nSimply placing the result on the last line will not work.\\n\\nArgs:\\n    query: The query to search for. Remember, this is a google query, so write your query as you would in Google.\\n", "parameters": {"properties": {"code": {"type": "string"}}, "required": ["code"], "type": "object"}}}\n{"type": "function", "function": {"name": "search_wikipedia", "description": "\\n\\"Search Wikipedia for the given query and return the results. Includes multiple document titles and body content. \\"\\n\\"Great for looking up people, places, and things. Less great (but still good) for current events.\\"\\n", "parameters": {"properties": {"query": {"type": "string"}}, "required": ["query"], "type": "object"}}}\n{"type": "function", "function": {"name": "research_complete", "description": "Invoke this once you are completed with your research. Include a brief handoff message (1-2 sentences) how confident you are that your research uncovered a complete answer. (You do not need to provide the answer itself; all the documents you found will be returned to the user for you.)", "parameters": {"properties": {"handoff_message": {"type": "string"}}, "required": ["handoff_message"], "type": "object", "additionalProperties": false}, "strict": true}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{"name": <function-name>, "arguments": <args-json-object>}\n</tool_call><|im_end|>\n<|im_start|>user\nWhat are the latest Magic: The Gathering sets?<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.2, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=31657, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=GuidedDecodingParams(json={'type': 'array', 'minItems': 1, 'items': {'type': 'object', 'anyOf': [{'properties': {'name': {'type': 'string', 'enum': ['search_web']}, 'parameters': {'properties': {'query': {'type': 'string'}}, 'required': ['query'], 'type': 'object'}}, 'required': ['name', 'parameters']}, {'properties': {'name': {'type': 'string', 'enum': ['visit_url']}, 'parameters': {'properties': {'url': {'type': 'string'}}, 'required': ['url'], 'type': 'object'}}, 'required': ['name', 'parameters']}, {'properties': {'name': {'type': 'string', 'enum': ['execute_python_code']}, 'parameters': {'properties': {'code': {'type': 'string'}}, 'required': ['code'], 'type': 'object'}}, 'required': ['name', 'parameters']}, {'properties': {'name': {'type': 'string', 'enum': ['search_wikipedia']}, 'parameters': {'properties': {'query': {'type': 'string'}}, 'required': ['query'], 'type': 'object'}}, 'required': ['name', 'parameters']}, {'properties': {'name': {'type': 'string', 'enum': ['research_complete']}, 'parameters': {'properties': {'handoff_message': {'type': 'string'}}, 'required': ['handoff_message'], 'type': 'object', 'additionalProperties': False}}, 'required': ['name', 'parameters']}]}}, regex=None, choice=None, grammar=None, json_object=None, backend=None, backend_was_auto=False, disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, whitespace_pattern=None, structural_tag=None), extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
[llmhost] | INFO 06-02 21:34:55 [async_llm.py:261] Added request chatcmpl-9060c008664b44d2843e47ac15fe420d.
[llmhost] | INFO:     10.89.1.1:36838 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request

[chaunceyjiang]

Can you help test this script?

from lmformatenforcer.external.jsonschemaobject import JsonSchemaObject

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description":
                        "The city to find the weather for, e.g. 'Vienna'",
                        "default": "Vienna",
                    },
                    "country": {
                        "type":
                        "string",
                        "description":
                        "The country that the city is in, e.g. 'Austria'",
                    },
                    "unit": {
                        "type": "string",
                        "description":
                        "The unit to fetch the temperature in",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["country", "unit"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_forecast",
            "description": "Get the weather forecast for a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description":
                        "The city to get the forecast for, e.g. 'Vienna'",
                        "default": "Vienna",
                    },
                    "country": {
                        "type":
                        "string",
                        "description":
                        "The country that the city is in, e.g. 'Austria'",
                    },
                    "days": {
                        "type":
                        "integer",
                        "description":
                        "Number of days to get the forecast for (1-7)",
                    },
                    "unit": {
                        "type": "string",
                        "description":
                        "The unit to fetch the temperature in",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["country", "days", "unit"],
            },
        },
    },
]

messages = [
    {
        "role": "user",
        "content": "Hi! How are you doing today?"
    },
    {
        "role": "assistant",
        "content": "I'm doing well! How can I help you?"
    },
    {
        "role":
        "user",
        "content":
        "Can you tell me what the current weather is in Berlin and the "\
        "forecast for the next 5 days, in fahrenheit?",
    },
]


chat_completion = client.chat.completions.create(
    messages=messages,
    model='',
    tools=tools,
    tool_choice="required"
)
print("Chat completion response:")
print(f"Chat completion: {chat_completion}")
for choice in chat_completion.choices:
    if choice.message.tool_calls:
        print(
            f"Tool calls: {choice.message.tool_calls}")
    else:
        print("No tool calls found.")
assert chat_completion.choices[0].message.tool_calls is not None
assert len(chat_completion.choices[0].message.tool_calls) > 0

[andysalerno]

Sure, here's the output from the script side:

openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': '1 validation error for list[function-wrap[__log_extra_fields__()]]\n  Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value=\'<think>\\nOkay, the user ..."unit": "fahrenheit"}}]\', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid', 'type': 'BadRequestError', 'param': None, 'code': 400}

Here's the relevant logs from vllm v0.9.0.1:

[llmhost] | INFO 06-02 23:05:11 [logger.py:42] Received request chatcmpl-e394fbf05847451d966cfbbbd5d49476: prompt: '<|im_start|>system\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{"type": "function", "function": {"name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city to find the weather for, e.g. \'Vienna\'", "default": "Vienna"}, "country": {"type": "string", "description": "The country that the city is in, e.g. \'Austria\'"}, "unit": {"type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"]}}, "required": ["country", "unit"]}}}\n{"type": "function", "function": {"name": "get_forecast", "description": "Get the weather forecast for a given location", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city to get the forecast for, e.g. \'Vienna\'", "default": "Vienna"}, "country": {"type": "string", "description": "The country that the city is in, e.g. \'Austria\'"}, "days": {"type": "integer", "description": "Number of days to get the forecast for (1-7)"}, "unit": {"type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"]}}, "required": ["country", "days", "unit"]}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{"name": <function-name>, "arguments": <args-json-object>}\n</tool_call><|im_end|>\n<|im_start|>user\nHi! How are you doing today?<|im_end|>\n<|im_start|>assistant\nI\'m doing well! How can I help you?<|im_end|>\n<|im_start|>user\nCan you tell me what the current weather is in Berlin and the forecast for the next 5 days, in fahrenheit?<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32174, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=GuidedDecodingParams(json={'type': 'array', 'minItems': 1, 'items': {'type': 'object', 'anyOf': [{'properties': {'name': {'type': 'string', 'enum': ['get_current_weather']}, 'parameters': {'type': 'object', 'properties': {'city': {'type': 'string', 'description': "The city to find the weather for, e.g. 'Vienna'", 'default': 'Vienna'}, 'country': {'type': 'string', 'description': "The country that the city is in, e.g. 'Austria'"}, 'unit': {'type': 'string', 'description': 'The unit to fetch the temperature in', 'enum': ['celsius', 'fahrenheit']}}, 'required': ['country', 'unit']}}, 'required': ['name', 'parameters']}, {'properties': {'name': {'type': 'string', 'enum': ['get_forecast']}, 'parameters': {'type': 'object', 'properties': {'city': {'type': 'string', 'description': "The city to get the forecast for, e.g. 'Vienna'", 'default': 'Vienna'}, 'country': {'type': 'string', 'description': "The country that the city is in, e.g. 'Austria'"}, 'days': {'type': 'integer', 'description': 'Number of days to get the forecast for (1-7)'}, 'unit': {'type': 'string', 'description': 'The unit to fetch the temperature in', 'enum': ['celsius', 'fahrenheit']}}, 'required': ['country', 'days', 'unit']}}, 'required': ['name', 'parameters']}]}}, regex=None, choice=None, grammar=None, json_object=None, backend=None, backend_was_auto=False, disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, whitespace_pattern=None, structural_tag=None), extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
[llmhost] | INFO 06-02 23:05:11 [async_llm.py:261] Added request chatcmpl-e394fbf05847451d966cfbbbd5d49476.
[llmhost] | INFO 06-02 23:05:15 [loggers.py:116] Engine 000: Avg prompt throughput: 47.4 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 26.2%
[llmhost] | INFO:     100.78.121.6:59470 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request

reminder, here are the args I am passing to vllm:

      --model Qwen/Qwen3-30B-A3B-FP8
      --reasoning-parser qwen3
      --max-num-seqs 4
      --tool-call-parser hermes
      --enable-auto-tool-choice
      --enable-reasoning
      --served-model-name model
      --guided-decoding-backend xgrammar
      --max-model-len 32648
      -tp 2

[andysalerno]

and btw, if I remove tool_choice="required", then it returns a valid response as expected, without a 400:

Chat completion response:
Chat completion: ChatCompletion(id='chatcmpl-fbabd9859a7848b98105a4c94210b0c1', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content='\n\n', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='chatcmpl-tool-da9c2f3db2d04789b17efa4c3cf7c2e8', function=Function(arguments='{"city": "Berlin", "country": "Germany", "unit": "fahrenheit"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='chatcmpl-tool-5681e3092fe74babae2e3a9a2e6666a0', function=Function(arguments='{"city": "Berlin", "country": "Germany", "days": 5, "unit": "fahrenheit"}', name='get_forecast'), type='function')], reasoning_content="\nOkay, the user is asking for the current weather in Berlin and the 5-day forecast, both in Fahrenheit. Let me check the tools provided.\n\nFirst, for the current weather, there's the get_current_weather function. It requires city, country, and unit. The user mentioned Berlin, but didn't specify the country. However, Berlin is a city in Germany, so I'll assume the country is Germany. The unit should be fahrenheit as requested.\n\nNext, the forecast. The get_forecast function needs city, country, days, and unit. The user wants the next 5 days, so days=5. Again, country is Germany, and unit is fahrenheit. I need to make sure the parameters are correctly set for both functions. Both functions are required here, so I'll generate two tool calls.\n"), stop_reason=None)], created=1748930801, model='model', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=245, prompt_tokens=474, total_tokens=719, completion_tokens_details=None, prompt_tokens_details=None), prompt_logprobs=None, kv_transfer_params=None)
Tool calls: [ChatCompletionMessageToolCall(id='chatcmpl-tool-da9c2f3db2d04789b17efa4c3cf7c2e8', function=Function(arguments='{"city": "Berlin", "country": "Germany", "unit": "fahrenheit"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='chatcmpl-tool-5681e3092fe74babae2e3a9a2e6666a0', function=Function(arguments='{"city": "Berlin", "country": "Germany", "days": 5, "unit": "fahrenheit"}', name='get_forecast'), type='function')]

[chaunceyjiang]

Indeed, I can reproduce this issue locally as well. I will try to fix it.

[chaunceyjiang]

@andysalerno Could you help test #19075? I’ve already fixed this issue.

[andysalerno]

@chaunceyjiang are there nightly docker images available? I just tried building the image myself on the latest commit, but my 3090 is too old to build some of the newer fp4 kernels or something, so it fails.