[Bug]: Mistral tool parser failed to parse function calling

[hahmad2008]

Your current environment

vllm version 0.8.3

🐛 Describe the bug

I used mistral parser to get the function calling from mistral3.1 -AWQ model. The output looks good as content text but vllm mistral parser failed to parse it as function calling output.

• command:

VLLM_USE_V1=0  vllm serve OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym \
                              --max-model-len 4096 --gpu-memory-utilization 0.8   --distributed-executor-backend ray  --dtype float16 \
                              --tool-call-parser mistral --enable-auto-tool-choice --chat-template tool_chat_template_mistral3.1_json.jinja

• Request:

 {
      "model": "OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym",
    "messages": [
        {
            "content": "What's the weather like in San Francisco in Celsius?",
            "role": "user"
        }
    ],
    "max_completion_tokens": 128,
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "City and state, e.g., 'San Francisco, CA'"
                        },
                        "unit": {
                            "type": "string",
                            "enum": [
                                "celsius",
                                "fahrenheit"
                            ]
                        }
                    },
                    "required": [
                        "location",
                        "unit"
                    ]
                }
            }
        }
    ],
    "tool_choice": "auto"
  }

• Response:

{
  "id": "chatcmpl-e57f4b2c6abe41ffb8175456b85d2b7a",
  "object": "chat.completion",
  "created": 1744036234,
  "model": "OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "reasoning_content": null,
        "content": "[{\"name\": \"get_weather\", \"arguments\": {\"location\": \"San Francisco\", \"unit\": \"celsius\"}}]",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null
    }
  ],
       ...
}

• tool_chat_template_mistral3.1_json.jinja
• taken from this repo

{%- set today = strftime_now("%Y-%m-%d") %}
{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information, you say that you don't have the information and don't make up anything.\nIf the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. \"What are some good restaurants around me?\" => \"Where are you?\" or \"When is the next flight to Tokyo\" => \"Where do you travel from?\")" %}

{{- bos_token }}

{%- if messages[0]['role'] == 'system' %}
    {%- set system_message = messages[0]['content'] %}
    {%- set loop_messages = messages[1:] %}
{%- else %}
    {%- set system_message = default_system_message %}
    {%- set loop_messages = messages %}
{%- endif %}
{%- set user_messages = loop_messages | selectattr('role', 'equalto', 'user') | list %}
{{- '[SYSTEM_PROMPT]' + system_message + '[/SYSTEM_PROMPT]' }}

{%- for message in loop_messages %}
    {%- if message['role'] == 'user' %}
        {%- if tools and (message == user_messages[-1]) %}
            {{- '[AVAILABLE_TOOLS][' }}
            {%- for tool in tools %}
                {%- set tool = tool.function %}
                {{- '{"type": "function", "function": {' }}
                {%- for key, val in tool.items() if key != 'return' %}
                    {{- key|tojson + ': ' + val|tojson }}
                    {%- if not loop.last %}
                        {{- ', ' }}
                    {%- endif %}
                {%- endfor %}
                {{- '}}' }}
                {%- if not loop.last %}
                    {{- ', ' }}
                {%- else %}
                    {{- ']' }}
                {%- endif %}
            {%- endfor %}
            {{- '[/AVAILABLE_TOOLS]' }}
        {%- endif %}
	    {%- if message['content'] is string %}
            {{- '[INST]' + message['content'] + '[/INST]' }}
	    {%- else %}
		    {{- '[INST]' }}
		    {%- for block in message['content'] %}
			    {%- if block['type'] == 'text' %}
				    {{- block['text'] }}
			    {%- elif block['type'] == 'image' or block['type'] == 'image_url' %}
				    {{- '[IMG]' }}
				{%- else %}
				    {{- raise_exception('Only text and image blocks are supported in message content!') }}
				{%- endif %}
			{%- endfor %}
		    {{- '[/INST]' }}
		{%- endif %}
    {%- elif message['role'] == 'system' %}
        {{- '[SYSTEM_PROMPT]' + message['content'] + '[/SYSTEM_PROMPT]' }}
    {%- elif message['role'] == 'assistant' %}
        {{- message.content if message.content }}
        {%- if message.tool_calls %}
            {{- '[TOOL_CALLS][' }}
            {%- for tool_call in message.tool_calls %}
                {%- set out = tool_call.function|tojson %}
                {{- out[:-1] }}
                {{- ', "id": ' + tool_call.id|tojson if tool_call.id }}
                {{- '}' }}
                {%- if not loop.last %}
                    {{- ', ' }}
                {%- else %}
                    {{- ']' }}
                {%- endif %}
            {%- endfor %}
        {%- endif %}
        {{- eos_token }}
    {%- elif message['role'] == 'tool' %}
        {%- if message.tool_call_id is undefined or message.tool_call_id is none %}
            {{- raise_exception("Tool call must have a valid tool_call_id!") }}
        {%- endif %}
        {%- if message.content and message.content.content is defined %}
            {%- set content = message.content.content %}
        {%- else %}
            {%- set content = message.content %}
        {%- endif %}
        {{- '[TOOL_RESULTS]' }}
        {{- message.tool_call_id }}
        {{- '[TOOL_CONTENT]' + (content if content is string else content|tojson) + '[/TOOL_RESULTS]' }}
    {%- else %}
        {{- raise_exception('Only user, tool, system and assistant roles are supported!') }}
    {%- endif %}
{%- endfor %}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[andysalerno]

I'm having the same issue.

It appears the problem is because the tool parser is expecting the json output to be prefixed by [TOOL_CALLS]

https://github.com/vllm-project/vllm/blob/3badb0213b11dcd74db85dea04b883f3391c47a7/vllm/entrypoints/openai/tool_parsers/mistral_tool_parser.py#L84C1-L88C70

My suspicion is, the model is indeed generating [TOOL_CALLS] before the tool json, but since [TOOL_CALLS] is a special token, and skip_special_tokens defaults to true, then the special tokens are stripped before the parser is triggered, so it does not match.

That's a total guess though.

[gcalmettes]

could you try adding "skip_special_tokens": false as extra_body in your request ?

if using a json payload, just add it to your json:

{
      "model": "OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym",
    ...
    "tool_choice": "auto",
    "skip_special_tokens": false
  }

if using the openAI client, add it as extra_body param:

response = client.chat.completions.create(
    model="your model",
    messages=[...],
    extra_body={
        "skip_special_tokens": False
    },
    tools=tools,
    tool_choice="auto"
)

`