[Feature]: Allow setting tool_choice="none" in LLM calls if the OpenAI comaptible vllm server is started with --enable-auto-tool-choice

[deheim]

🚀 The feature, motivation and pitch

Apparently, when starting the (OpenAI compatible) vllm server with the option --enable-auto-tool-choice to handle function calls, it is not possible to prevent tool calls when calling the model.

According to the OpenAI API, this should be possible by setting tool_choice='none'. However, setting tool_choice to "none" seems to be not yet supported by vllm.

Currently, an ValueError is thrown:
Call:

client=OpenAI(...)
completion = client.chat.completions.create(
    messages=[
        {"role": "user", "content": "Write a poem"}],
    model="meta-llama-3.1",
    tool_choice='none'
)

Error Message:
openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "[{'type': 'value_error', 'loc': ('body',), 'msg': 'Value error, When using `tool_choice`, `tools` must be set.', 'input': {'messages': [{'role': 'user', 'content': 'Write a poem'}], 'model': 'meta-llama-3.1', 'tool_choice': 'none'}, 'ctx': {'error': ValueError('When using `tool_choice`, `tools` must be set.')}}]", 'type': 'BadRequestError', 'param': None, 'code': 400}

Alternatives

The only alternative would be hosting two models in parallel. One for function calls and one for getting outputs without function calls which is not really practicable.

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[laserkelvin]

There's since been another related issue, #10526.

IMO the current issue and #10526 are not really feature requests but more bugs: trying to use vLLM served models for downstream applications is essentially incompatible when the same functionality works seamlessly on other serving frameworks like ollama. When agents are allowed to choose to not use a tool, as per the OpenAI spec, we need this supported so that workflows/systems don't break when trying to migrate from ollama to vLLM.

I'm not sure what would get this issue triaged - looking at #8568 it doesn't seem like it would take a lot. Would the maintainers be open to a PR for this?

[laserkelvin]

Actually, I think I've made a mistake: #10000 seems to actually address the issue @deheim outlined, and the exception message being returned by vLLM as of 0.6.6.post1 reflects this:

'Value error, `tool_choice` must either be a named tool, "auto", or "none".

So nominally "none" is supported. My previous comment stands in that tool usage is still not up to OpenAI API specification (I actually need "required", as mentioned in #10526, not "none"). I'm leaving the earlier comment in case others stumble on this.

[deheim]

My initial post refered to Version v0.6.3. Maybe already some changes have been made such that tool_choice "none" is now already supported. I can try it out later.

Having the option to set it to "required" would be, however, also nice, I agree.

But yes maybe the classification as a bug is more appropriate since the error message for other values for tool choice explicitly says that tool_choice none is a valid value. I will update it after my tests with the most recent vllm version