[Frontend][Bug Fix] Update llama4 pythonic jinja template and llama4_pythonic parser

[wukaixingxp]

Change the llama4 pythonic template and small fix on the edge case where llama4 model may output <|python_start|> unexpectedly.
BFCL test result:

Name	reported	Base_vllm	Pythonic_vllm
Model	Llama-4-Scout-17B-16E-Instruct (FC)	Llama-4-Scout-17B-16E-Instruct (FC)	Llama-4-Scout-17B-16E-Instruct (FC)
Overall Acc	45.41%	47.78%	55.97%
Non-Live AST Acc	83.48%	83.67%	80.04%
Non-Live Simple AST	79.42%	79.17%	78.67%
Non-Live Multiple AST	95%	94.00%	92.00%
Non-Live Parallel AST	81.5%	80.50%	77.00%
Non-Live Parallel Multiple AST	78%	81.00%	72.50%
Live Acc	57.97%	58.69%	74.06%
Live Simple AST	77.91%	82.17%	80.62%
Live Multiple AST	74.36%	74.07%	73.22%
Live Parallel AST	68.75%	75.00%	75.00%
Live Parallel Multiple AST	62.5%	70.83%	66.67%
Multi Turn Acc	1.88%	6.38%	13.00%
Multi Turn Base	2%	8.50%	15.50%
Multi Turn Miss Func	2.5%	5.00%	14.00%
Multi Turn Miss Param	1%	4.00%	11.00%
Multi Turn Long Context	2%	8.00%	11.50%
Relevance Detection	100%	94.44%	77.78%
Irrelevance Detection	39.66%	44.38%	78.70%

NOTE: Since BFCL has a default tool-call system prompt, we need to manually modified pythonic and json system prompt from here
For jinja template please see this example:
Given this test data:

{
    "bos_token": "<|begin_of_text|>",
    "add_generation_prompt": true,
 "custom_tools": [
    {
        "name": "get_weather",
        "description": "Get weather info for places",
        "parameters": {
            "type": "dict",
            "required": ["city"],
            "properties": {
                "city": {
                    "type": "string",
                    "description": "The name of the city to get the weather for"
                },
                "metric": {
                    "type": "string",
                    "description": "The metric for weather. Options are: celsius, fahrenheit",
                    "default": "celsius"
                }
            }
        }
    }
],
    "messages": [
#{"role": "system", "content": "you are helpful assistant"},

{"role": "user", "content": "Who are you?"},
{"role": "assistant", "content": "I am a Llama 4 model",tool_calls:[]},
      {"role": "user", "content": "What is the weather in SF and Seattle?"},
{"role": "assistant", "content": "[get_weather(city=\"San Francisco\"), get_weather(city=\"Seattle\")]",tool_calls:[]},
{
    "role": "ipython",
    "content": "[        {            \"response\": \"Sunny 75\"        },        {            \"response\": \"Rainy 65\"        },    ]"
},{"role": "assistant", "content": "SF is Sunny and Seattle is Rainy"},
{"role": "user", "content": "What is the weather in NYC and SF"},
{"role": "assistant", "content": "",'tool_calls': [
            {
                'name': 'get_weather',
                'arguments': {
                    'city': 'NYC',
                   'metric': 'fahrenheit',
                }
            },
{
                'name': 'get_weather',
                'arguments': {
                    'city': 'SF',
                   'metric': 'fahrenheit',
                }
            },
        ],},
    ]
  }

The jinja template will render the output like this:

<|begin_of_text|><|header_start|>system<|header_end|>

You are a helpful assistant and an expert in function composition. You can answer general questions using your internal knowledge OR invoke functions when necessary. Follow these strict guidelines:

1. FUNCTION CALLS:
- ONLY use functions that are EXPLICITLY listed in the function list below
- If NO functions are listed (empty function list []), respond ONLY with internal knowledge or "I don't have access to [Unavailable service] information"
- If a function is not in the list, respond ONLY with internal knowledge or "I don't have access to [Unavailable service] information"
- If ALL required parameters are present AND the query EXACTLY matches a listed function's purpose: output ONLY the function call(s)
- Use exact format: [func_name1(param1=value1, param2=value2), func_name2(...)]
Examples:
CORRECT: [get_weather(location="Vancouver"), calculate_route(start="Boston", end="New York")] <- Only if get_weather and calculate_route are in function list
INCORRECT: get_weather(location="New York")
INCORRECT: Let me check the weather: [get_weather(location="New York")]
INCORRECT: [get_events(location="Singapore")] <- If function not in list

2. RESPONSE RULES:
- For pure function requests matching a listed function: ONLY output the function call(s)
- For knowledge questions: ONLY output text
- For missing parameters: ONLY request the specific missing parameters
- For unavailable services (not in function list): output ONLY with internal knowledge or "I don't have access to [Unavailable service] information". Do NOT execute a function call.
- If the query asks for information beyond what a listed function provides: output ONLY with internal knowledge about your limitations
- NEVER combine text and function calls in the same response
- NEVER suggest alternative functions when the requested service is unavailable
- NEVER create or invent new functions not listed below

3. STRICT BOUNDARIES:
- ONLY use functions from the list below - no exceptions
- NEVER use a function as an alternative to unavailable information
- NEVER call functions not present in the function list
- NEVER add explanatory text to function calls
- NEVER respond with empty brackets
- Use proper Python/JSON syntax for function calls
- Check the function list carefully before responding

4. TOOL RESPONSE HANDLING:
- When receiving tool responses: provide concise, natural language responses
- Don't repeat tool response verbatim
- Don't add supplementary information

Here is a list of functions in JSON format that you can invoke:
[
    {
        "description": "Get weather info for places",
        "name": "get_weather",
        "parameters": {
            "properties": {
                "city": {
                    "description": "The name of the city to get the weather for",
                    "type": "string"
                },
                "metric": {
                    "default": "celsius",
                    "description": "The metric for weather. Options are: celsius, fahrenheit",
                    "type": "string"
                }
            },
            "required": [
                "city"
            ],
            "type": "dict"
        }
    }
]
<|eot|><|header_start|>user<|header_end|>

Who are you?<|eot|><|header_start|>assistant<|header_end|>

I am a Llama 4 model<|eot|><|header_start|>user<|header_end|>

What is the weather in SF and Seattle?<|eot|><|header_start|>assistant<|header_end|>

[get_weather(city="San Francisco"), get_weather(city="Seattle")]<|eot|><|header_start|>ipython<|header_end|>

"[        {            \"response\": \"Sunny 75\"        },        {            \"response\": \"Rainy 65\"        },    ]"<|eot|><|header_start|>assistant<|header_end|>

SF is Sunny and Seattle is Rainy<|eot|><|header_start|>user<|header_end|>

What is the weather in NYC and SF<|eot|><|header_start|>assistant<|header_end|>

[get_weather(city="NYC", metric="fahrenheit"), get_weather(city="SF", metric="fahrenheit")]<|eot|><|header_start|>assistant<|header_end|>

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[yeqcharlotte]

great thank you! please fix the long lines.

and do could you get the results from llama-stack eval?
please also double check unit tests on vllm side pytest -s -vv tests/tool_use --models llama4 --extended

[yeqcharlotte]

thanks for sharing the new template that reaches BFCL parity! mind also update the test summary to make it readable?

[FWao]

The new Jinja template behaves differently from the old one when a user provides a system prompt.

Problem:
In the new template, if the user sets a system message, the list of tools is added—but the instructions for using Python-style tool calls are missing. Because of this, the model makes JSON-style tool calls instead.

Expected (Old Template):
In the old template, tool instructions were always included, even if the user provided a custom system prompt. The model worked as expected.

Current (New Template):
Now, the user has to manually add tool call instructions to the system prompt. This was not needed before.

[wukaixingxp]

The new Jinja template behaves differently from the old one when a user provides a system prompt.

Problem: In the new template, if the user sets a system message, the list of tools is added—but the instructions for using Python-style tool calls are missing. Because of this, the model makes JSON-style tool calls instead.

Expected (Old Template): In the old template, tool instructions were always included, even if the user provided a custom system prompt. The model worked as expected.

Current (New Template): Now, the user has to manually add tool call instructions to the system prompt. This was not needed before.

Thank you so much for this feedback.. I will write more test to make sure it works for all other cases..

[wukaixingxp]

Hi! Thanks for your feedback. The minimum tool-definition string and tool-output expectation will be appended to the user-provided system prompt now. For tool-call, the idea is that for basic users, we recommend not to set any system prompt so that our default comprehensive system prompt will be used, whereas for advanced users, who want to use their own customized system prompt, we will provide minimum tool-definition string and tool-output expectation at the end. Please check the following test examples:
No user-provided system prompt, use default
User-provided system prompt, append tool definition at the end
User-provided system prompt, but no-tool provided
No system prompt and no tool

The new Jinja template behaves differently from the old one when a user provides a system prompt.

Problem: In the new template, if the user sets a system message, the list of tools is added—but the instructions for using Python-style tool calls are missing. Because of this, the model makes JSON-style tool calls instead.

Expected (Old Template): In the old template, tool instructions were always included, even if the user provided a custom system prompt. The model worked as expected.

Current (New Template): Now, the user has to manually add tool call instructions to the system prompt. This was not needed before.

[bbrowning]

Attemping to run these changes locally with the Berkeley function calling leaderboard and my own vLLM, it appears to only be using the completions endpoint (instead of chat completions) when testing the Llama 4 Scout model. For completeness, here's how I'm running bfcl against my local vLLM serving Llama 4 Scout:

bfcl generate --model meta-llama/Llama-4-Scout-17B-16E-Instruct-FC --skip-server-setup

The configuration for meta-llama/Llama-4-Scout-17B-16E-Instruct-FC in bfcl sends all requests to the completions endpoint (instead of chat/completions), so the jinja template and tool call parser are not used at all in that flow. Are there changes used to update bfcl to use chat/completions when testing Llama 4 Scout?

[wukaixingxp]

Attemping to run these changes locally with the Berkeley function calling leaderboard and my own vLLM, it appears to only be using the completions endpoint (instead of chat completions) when testing the Llama 4 Scout model. For completeness, here's how I'm running bfcl against my local vLLM serving Llama 4 Scout:

bfcl generate --model meta-llama/Llama-4-Scout-17B-16E-Instruct-FC --skip-server-setup

The configuration for meta-llama/Llama-4-Scout-17B-16E-Instruct-FC in bfcl sends all requests to the completions endpoint (instead of chat/completions), so the jinja template and tool call parser are not used at all in that flow. Are there changes used to update bfcl to use chat/completions when testing Llama 4 Scout?

Change this default system prompt to

You are a helpful assistant and an expert in function composition. You can answer general questions using your internal knowledge OR invoke functions when necessary. Follow these strict guidelines:

1. FUNCTION CALLS:
- ONLY use functions that are EXPLICITLY listed in the function list below
- If NO functions are listed (empty function list []), respond ONLY with internal knowledge or "I don't have access to [Unavailable service] information"
- If a function is not in the list, respond ONLY with internal knowledge or "I don't have access to [Unavailable service] information"
- If ALL required parameters are present AND the query EXACTLY matches a listed function's purpose: output ONLY the function call(s)
- Use exact format: [func_name1(param1=value1, param2=value2), func_name2(...)]
Examples:
CORRECT: [get_weather(location="Vancouver"), calculate_route(start="Boston", end="New York")] <- Only if get_weather and calculate_route are in function list
INCORRECT: get_weather(location="New York")
INCORRECT: Let me check the weather: [get_weather(location="New York")]
INCORRECT: [get_events(location="Singapore")] <- If function not in list

2. RESPONSE RULES:
- For pure function requests matching a listed function: ONLY output the function call(s)
- For knowledge questions: ONLY output text
- For missing parameters: ONLY request the specific missing parameters
- For unavailable services (not in function list): output ONLY with internal knowledge or "I don't have access to [Unavailable service] information". Do NOT execute a function call.
- If the query asks for information beyond what a listed function provides: output ONLY with internal knowledge about your limitations
- NEVER combine text and function calls in the same response
- NEVER suggest alternative functions when the requested service is unavailable
- NEVER create or invent new functions not listed below

3. STRICT BOUNDARIES:
- ONLY use functions from the list below - no exceptions
- NEVER use a function as an alternative to unavailable information
- NEVER call functions not present in the function list
- NEVER add explanatory text to function calls
- NEVER respond with empty brackets
- Use proper Python/JSON syntax for function calls
- Check the function list carefully before responding

4. TOOL RESPONSE HANDLING:
- When receiving tool responses: provide concise, natural language responses
- Don't repeat tool response verbatim
- Don't add supplementary information

[bbrowning]

Ahh, I see. I was trying to run the function calling leaderboard via vLLM in a way that exercised the actual vLLM jinja template and tool parser. But, I see what you're doing is not doing that and instead just copying the same prompt into the bfcl code but still using the completions endpoint for testing. Your way is a reasonable way to test the prompt, although that won't end up actually exercising this tool call parser change at all I don't think?

[wukaixingxp]

Yeah.. I think the function definition are prepared from BFCL to the system prompt to bypass our jinja template/ parser. I mentioned BFCL just to show this jinja template can give a better result. I will run pytest -s -vv tests/tool_use --models llama4 --extended and llama-stack-evals to test the jinja_template + parser in my PRs.

[wukaixingxp]

Ahh, I see. I was trying to run the function calling leaderboard via vLLM in a way that exercised the actual vLLM jinja template and tool parser. But, I see what you're doing is not doing that and instead just copying the same prompt into the bfcl code but still using the completions endpoint for testing. Your way is a reasonable way to test the prompt, although that won't end up actually exercising this tool call parser change at all I don't think?

BTW, llama-stack-evals also support BFCL now, I think it will just take a openai compatible server and rely on the jinja template/parser from vllm.

[wukaixingxp]

Test with llama-stack-eval

1. create ./llama4_pythonic.jinja based on the jinja template
2. install current commit vllm uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly and modified the site_package pythonic_tool_parser.py in /home/.conda/envs/vllm/lib/python3.10/sitepackages/vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py
3. Started the vllm server VLLM_DISABLE_COMPILE_CACHE=1 vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct -tp4 --seed 0 --max-model-len=100000 --host 0.0.0.0 --port 8001 --enable-auto-tool-choice --tool-call-parser llama4_pythonic --chat-template ./llama4_pythonic.jinja --limit-mm-per-prompt image=5 --max-num-seqs 20
4. Install llama-stack evals: git clone https://github.com/fairinternal/llama-stack-evals.git; cd llama-stack-evals;pip install -e .
5. run llama-stack-evals: llama-stack-evals run-tests --model meta-llama/Llama-4-Scout-17B-16E-Instruct --provider vllm
6. results, only two error code test failed, all other test passed:

llama-stack-evals run-tests --model meta-llama/Llama-4-Scout-17B-16E-Instruct --provider vllm

=== Running tests for provider: vllm, model: meta-llama/Llama-4-Scout-17B-16E-Instruct ===
Running command: /home/kaiwu/.conda/envs/evals/bin/python3.12 -m pytest llama_stack_evals/functional_tests/openai_api/test_chat_completion.py --model=meta-llama/Llama-4-Scout-17B-16E-Instruct -v --provider=vllm
====================================== test session starts ======================================
platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.6.0 -- /home/kaiwu/.conda/envs/evals/bin/python3.12
cachedir: .pytest_cache
metadata: {'Python': '3.12.9', 'Platform': 'Linux-6.4.3-0_fbk20_zion_2830_g3e5ab162667d-x86_64-with-glibc2.34', 'Packages': {'pytest': '8.3.5', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.9.0', 'metadata': '3.1.1', 'json-report': '1.5.0'}}
rootdir: /home/kaiwu/work/llama-stack-evals
configfile: pyproject.toml
plugins: anyio-4.9.0, metadata-3.1.1, json-report-1.5.0
collected 34 items                                                                              

llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_basic[meta-llama/Llama-4-Scout-17B-16E-Instruct-earth] PASSED [  2%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_basic[meta-llama/Llama-4-Scout-17B-16E-Instruct-saturn] PASSED [  5%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_basic[meta-llama/Llama-4-Scout-17B-16E-Instruct-earth] PASSED [  8%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_basic[meta-llama/Llama-4-Scout-17B-16E-Instruct-saturn] PASSED [ 11%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_image[meta-llama/Llama-4-Scout-17B-16E-Instruct-case0] PASSED [ 14%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_image[meta-llama/Llama-4-Scout-17B-16E-Instruct-case0] PASSED [ 17%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_structured_output[meta-llama/Llama-4-Scout-17B-16E-Instruct-extract] PASSED [ 20%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_structured_output[meta-llama/Llama-4-Scout-17B-16E-Instruct-extract] PASSED [ 23%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_tool_calling[meta-llama/Llama-4-Scout-17B-16E-Instruct-basic] PASSED [ 26%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_tool_calling[meta-llama/Llama-4-Scout-17B-16E-Instruct-user_provided_system_prompt] XFAIL [ 29%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_tool_calling[meta-llama/Llama-4-Scout-17B-16E-Instruct-array_param] XFAIL [ 32%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_tool_calling[meta-llama/Llama-4-Scout-17B-16E-Instruct-basic] PASSED [ 35%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_tool_calling[meta-llama/Llama-4-Scout-17B-16E-Instruct-user_provided_system_prompt] XFAIL [ 38%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_tool_calling[meta-llama/Llama-4-Scout-17B-16E-Instruct-array_param] XFAIL [ 41%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_tool_choice_required[meta-llama/Llama-4-Scout-17B-16E-Instruct-basic] XPASS [ 44%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_tool_choice_required[meta-llama/Llama-4-Scout-17B-16E-Instruct-basic] XPASS [ 47%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_tool_choice_none[meta-llama/Llama-4-Scout-17B-16E-Instruct-basic] XFAIL [ 50%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_tool_choice_none[meta-llama/Llama-4-Scout-17B-16E-Instruct-basic] XPASS [ 52%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_multi_turn_tool_calling[meta-llama/Llama-4-Scout-17B-16E-Instruct-text_then_tool] PASSED [ 55%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_multi_turn_tool_calling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tool_then_text] PASSED [ 58%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_multi_turn_tool_calling[meta-llama/Llama-4-Scout-17B-16E-Instruct-text_then_tool] XPASS [ 61%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_multi_turn_tool_calling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tool_then_text] XPASS [ 64%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_multi_turn_multiple_images[meta-llama/Llama-4-Scout-17B-16E-Instruct-stream=False] PASSED [ 67%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_multi_turn_multiple_images[meta-llama/Llama-4-Scout-17B-16E-Instruct-stream=True] PASSED [ 70%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-messages_missing] PASSED [ 73%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-messages_role_invalid] PASSED [ 76%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tool_choice_invalid] PASSED [ 79%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tool_choice_no_tools] PASSED [ 82%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid] FAILED [ 85%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-messages_missing] PASSED [ 88%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-messages_role_invalid] PASSED [ 91%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tool_choice_invalid] PASSED [ 94%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tool_choice_no_tools] PASSED [ 97%]
llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid] FAILED [100%]

=========================================== FAILURES ============================================
_ test_chat_non_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid] _

request = <FixtureRequest for <Function test_chat_non_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid]>>
openai_client = <openai.OpenAI object at 0x7fef7f10daf0>
model = 'meta-llama/Llama-4-Scout-17B-16E-Instruct', provider = 'vllm'
verification_config = {'cerebras': ProviderConfig(provider='cerebras', base_url='https://api.cerebras.ai/v1', api_key_var='CEREBRAS_API_KEY'...', 'Llama-4-Maverick-17B-128E-Instruct-FP8': 'Llama-4-Maverick-Instruct'}, test_exclusions={}, self_hosted=False), ...}
case = {'case_id': 'tools_type_invalid', 'input': {'messages': [{'content': 'Which planet do humans live on?', 'role': 'user'}], 'tools': [{'type': 'invalid'}]}, 'output': {'error': {'status_code': 400}}}

    @pytest.mark.parametrize(
        "case",
        chat_completion_test_cases["test_chat_input_validation"]["test_params"]["case"],
        ids=case_id_generator,
    )
    def test_chat_non_streaming_error_handling(request, openai_client, model, provider, verification_config, case):
        test_name_base = get_base_test_name(request)
        if should_skip_test(verification_config, provider, model, test_name_base):
            pytest.skip(f"Skipping {test_name_base} for model {model} on provider {provider} based on config.")
    
        with pytest.raises(APIError) as e:
            openai_client.chat.completions.create(
                model=model,
                messages=case["input"]["messages"],
                stream=False,
                tool_choice=case["input"]["tool_choice"] if "tool_choice" in case["input"] else None,
                tools=case["input"]["tools"] if "tools" in case["input"] else None,
            )
>       assert case["output"]["error"]["status_code"] == e.value.status_code
E       AssertionError: assert 400 == 500
E        +  where 500 = InternalServerError('Error code: 500').status_code
E        +    where InternalServerError('Error code: 500') = <ExceptionInfo InternalServerError('Error code: 500') tblen=5>.value

llama_stack_evals/functional_tests/openai_api/test_chat_completion.py:686: AssertionError
_ test_chat_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid] _

request = <FixtureRequest for <Function test_chat_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid]>>
openai_client = <openai.OpenAI object at 0x7fef7ee5eed0>
model = 'meta-llama/Llama-4-Scout-17B-16E-Instruct', provider = 'vllm'
verification_config = {'cerebras': ProviderConfig(provider='cerebras', base_url='https://api.cerebras.ai/v1', api_key_var='CEREBRAS_API_KEY'...', 'Llama-4-Maverick-17B-128E-Instruct-FP8': 'Llama-4-Maverick-Instruct'}, test_exclusions={}, self_hosted=False), ...}
case = {'case_id': 'tools_type_invalid', 'input': {'messages': [{'content': 'Which planet do humans live on?', 'role': 'user'}], 'tools': [{'type': 'invalid'}]}, 'output': {'error': {'status_code': 400}}}

    @pytest.mark.parametrize(
        "case",
        chat_completion_test_cases["test_chat_input_validation"]["test_params"]["case"],
        ids=case_id_generator,
    )
    def test_chat_streaming_error_handling(request, openai_client, model, provider, verification_config, case):
        test_name_base = get_base_test_name(request)
        if should_skip_test(verification_config, provider, model, test_name_base):
            pytest.skip(f"Skipping {test_name_base} for model {model} on provider {provider} based on config.")
    
        with pytest.raises(APIError) as e:
            response = openai_client.chat.completions.create(
                model=model,
                messages=case["input"]["messages"],
                stream=True,
                tool_choice=case["input"]["tool_choice"] if "tool_choice" in case["input"] else None,
                tools=case["input"]["tools"] if "tools" in case["input"] else None,
            )
            for _chunk in response:
                pass
>       assert str(case["output"]["error"]["status_code"]) in e.value.message
E       AssertionError: assert '400' in 'Error code: 500'
E        +  where '400' = str(400)
E        +  and   'Error code: 500' = InternalServerError('Error code: 500').message
E        +    where InternalServerError('Error code: 500') = <ExceptionInfo InternalServerError('Error code: 500') tblen=5>.value

llama_stack_evals/functional_tests/openai_api/test_chat_completion.py:709: AssertionError
==================================== short test summary info ====================================
FAILED llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid] - AssertionError: assert 400 == 500
FAILED llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid] - AssertionError: assert '400' in 'Error code: 500'
====================== 2 failed, 22 passed, 5 xfailed, 5 xpassed in 40.78s ======================

[wukaixingxp]

Tested using pytest -s -vv tests/tool_use --models llama4 --extended by @yeqcharlotte, only two errors but they are from model not from our code:

1. call get_weather instead of get_current_weather

>       assert tool_calls[0].function.name == WEATHER_TOOL["function"]["name"]
E       AssertionError: assert 'get_weather' == 'get_current_weather'
E         
E         - get_current_weather
E         + get_weather

tests/tool_use/test_tool_calls.py:41: AssertionError

2. Call get_weather when there is no tool:

>       assert choice.finish_reason != "tool_calls"  # "stop" or "length"
E       assert 'tool_calls' != 'tool_calls'
E        +  where 'tool_calls' = Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content=None, refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='chatcmpl-tool-09cfbb311c784573ab7cc83d14be7eb8', function=Function(arguments='{"city": "Dallas", "state": "TX", "unit": "fahrenheit"}', name='get_current_weather'), type='function')], reasoning_content=None), stop_reason=None).finish_reason

tests/tool_use/test_tool_calls.py:154: AssertionError

[DarkLight1337]

cc @aarnphm @houseroad

[aarnphm]

Can you also fix the pre-commit problem here?

few questions, but in general this looks good. Thanks for adding this.

[wukaixingxp]

Can you also fix the pre-commit problem here?

few questions, but in general this looks good. Thanks for adding this.

Yes.. I just fixed it. Now waiting for the final PR run to be completed

[yeqcharlotte]

thanks for adding the doc and unit tests!

[DarkLight1337]

We have temporarily paused all non-essential PRs to fix the CI. Please merge from main after #18418 is resolved

[wukaixingxp]

We have temporarily paused all non-essential PRs to fix the CI. Please merge from main after #18418 is resolved

@DarkLight1337 Can we me this PR now given that the issue has been fixed? CC: @yeqcharlotte @houseroad

[houseroad]

Can we rebase to main?

[bbrowning]

Testing this chat template locally, there is a logic bug here that results in the tool_definition never making its way into the actual prompt. The tools from the ChatCompletion request come in as a tools variable, but we only set the tool_definition value if there's a custom_tools value passed in.

The line {%- set tool_definition = tool_definition ~ (tools | tojson(indent=4)) %} needs to move outside of this if statement block, and should happen after we check and set tools to none if not defined. Here's how the first few lines should look:

{{- bos_token }}
{%- if custom_tools is defined and custom_tools%}
    {%- set tools = custom_tools %}
{%- endif %}
{%- if not tools is defined %}
    {%- set tools = none %}
{%- endif %}
{%- set tool_definition = tool_definition ~ (tools | tojson(indent=4)) %}

Without this change, the actual function definitions never get inserted into the model's prompt which results in it failing the majority of the bfclv3-api tests from llama-stack-evals repo.

[wukaixingxp]

Thanks for testing. You are right.. fixing this now!

[wukaixingxp]

Thanks to @bbrowning help, I just fixed a bug on the template, now tested with llama-stack-eval

================================================================ FAILURES =================================================================
__________________ test_chat_non_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid] ___________________

request = <FixtureRequest for <Function test_chat_non_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid]>>
openai_client = <openai.OpenAI object at 0x7fe732426f90>, model = 'meta-llama/Llama-4-Scout-17B-16E-Instruct', provider = 'vllm'
verification_config = {'cerebras': ProviderConfig(provider='cerebras', base_url='https://api.cerebras.ai/v1', api_key_var='CEREBRAS_API_KEY'...verick-17B-128E-Instruct-FP8', canonical_id='Llama-4-Maverick-Instruct')], test_exclusions={}, self_hosted=False), ...}
case = {'case_id': 'tools_type_invalid', 'input': {'messages': [{'content': 'Which planet do humans live on?', 'role': 'user'}], 'tools': [{'type': 'invalid'}]}, 'output': {'error': {'status_code': 400}}}

    @pytest.mark.parametrize(
        "case",
        chat_completion_test_cases["test_chat_input_validation"]["test_params"]["case"],
        ids=case_id_generator,
    )
    def test_chat_non_streaming_error_handling(request, openai_client, model, provider, verification_config, case):
        test_name_base = get_base_test_name(request)
        if should_skip_test(verification_config, provider, model, test_name_base):
            pytest.skip(f"Skipping {test_name_base} for model {model} on provider {provider} based on config.")
    
        with pytest.raises(APIError) as e:
            openai_client.chat.completions.create(
                model=model,
                messages=case["input"]["messages"],
                stream=False,
                tool_choice=case["input"]["tool_choice"] if "tool_choice" in case["input"] else None,
                tools=case["input"]["tools"] if "tools" in case["input"] else None,
            )
>       assert case["output"]["error"]["status_code"] == e.value.status_code
E       AssertionError: assert 400 == 500
E        +  where 500 = InternalServerError('Error code: 500').status_code
E        +    where InternalServerError('Error code: 500') = <ExceptionInfo InternalServerError('Error code: 500') tblen=5>.value

llama_stack_evals/functional_tests/openai_api/test_chat_completion.py:686: AssertionError
____________________ test_chat_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid] _____________________

request = <FixtureRequest for <Function test_chat_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid]>>
openai_client = <openai.OpenAI object at 0x7fe73258d6d0>, model = 'meta-llama/Llama-4-Scout-17B-16E-Instruct', provider = 'vllm'
verification_config = {'cerebras': ProviderConfig(provider='cerebras', base_url='https://api.cerebras.ai/v1', api_key_var='CEREBRAS_API_KEY'...verick-17B-128E-Instruct-FP8', canonical_id='Llama-4-Maverick-Instruct')], test_exclusions={}, self_hosted=False), ...}
case = {'case_id': 'tools_type_invalid', 'input': {'messages': [{'content': 'Which planet do humans live on?', 'role': 'user'}], 'tools': [{'type': 'invalid'}]}, 'output': {'error': {'status_code': 400}}}

    @pytest.mark.parametrize(
        "case",
        chat_completion_test_cases["test_chat_input_validation"]["test_params"]["case"],
        ids=case_id_generator,
    )
    def test_chat_streaming_error_handling(request, openai_client, model, provider, verification_config, case):
        test_name_base = get_base_test_name(request)
        if should_skip_test(verification_config, provider, model, test_name_base):
            pytest.skip(f"Skipping {test_name_base} for model {model} on provider {provider} based on config.")
    
        with pytest.raises(APIError) as e:
            response = openai_client.chat.completions.create(
                model=model,
                messages=case["input"]["messages"],
                stream=True,
                tool_choice=case["input"]["tool_choice"] if "tool_choice" in case["input"] else None,
                tools=case["input"]["tools"] if "tools" in case["input"] else None,
            )
            for _chunk in response:
                pass
>       assert str(case["output"]["error"]["status_code"]) in e.value.message
E       AssertionError: assert '400' in 'Error code: 500'
E        +  where '400' = str(400)
E        +  and   'Error code: 500' = InternalServerError('Error code: 500').message
E        +    where InternalServerError('Error code: 500') = <ExceptionInfo InternalServerError('Error code: 500') tblen=5>.value

llama_stack_evals/functional_tests/openai_api/test_chat_completion.py:709: AssertionError
========================================================= short test summary info =========================================================
FAILED llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_non_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid] - AssertionError: assert 400 == 500
FAILED llama_stack_evals/functional_tests/openai_api/test_chat_completion.py::test_chat_streaming_error_handling[meta-llama/Llama-4-Scout-17B-16E-Instruct-tools_type_invalid] - AssertionError: assert '400' in 'Error code: 500'
=========================================== 2 failed, 22 passed, 5 xfailed, 5 xpassed in 30.94s ===========================================
Tests failed for provider=vllm, model=meta-llama/Llama-4-Scout-17B-16E-Instruct with exit code 1