Function calling for OpenAI backend

[Yiyun-Liang]

Adding skeleton code for function calling with Open API models.
Example output (when tool_choice is "auto" or "required"):

system : You are a helpful assistant.
user : What's the weather like in San Francisco, Tokyo, Paris, and Beijing?
assistant : The current weather in San Francisco is 72°F, in Tokyo it is 10°C, and in Paris it is 22°C. Unfortunately, I couldn't retrieve the weather information for Beijing at the moment.

Example output (when tool_choice is "none"):

system : You are a helpful assistant.
user : What's the weather like in San Francisco, Tokyo, Paris, and Beijing?
assistant : I'm sorry for the inconvenience, but as an AI model developed by OpenAI, I don't have real-time capabilities to provide the current weather in San Francisco, Tokyo, Paris, and Beijing. I recommend using a reliable weather forecast service or website to get real-time weather updates for these cities.

The current implementation does not support:

• chat models not listed by OpenAI as of 07/25/24.
• non-chat models
• speculative execution

[Ying1123]

Thanks for the PR! I left a few comments. The review is still in progress.

[Ying1123]

use dictionary order instead.

[Yiyun-Liang]

Removing this given we are moving it to be part of SglGen.

[Ying1123]

We may also want to retrieve the results from function call.
Add tests for state["func_call_1"] in the function single().

[Ying1123]

s.messages_ should be updated after function call finished rather than in generate, and the append logic should happen in interpreter.py, see _execute_role_end() as a reference.

Additionally, changes prompt implicitly changes s.messages_. This is not safe. Changes s.messages_ then set prompt = s.messages_ is better.

[Yiyun-Liang]

Restructured the code a little bit based on your suggestions (with some minor tweaks but I can update if you think it's still better to move the function call generate call outside of generate (we will just have a simpler generate call):

Within openai.py

• build_function_call_messages(): a new function which builds function call messages. Given function signature is specific to open ai models, keeping the logic to parse inputs and produce function call messages within the backend code.
• generate(): Given prompt is local to the generate() call, I directly added function_call_messages to it so that we can call with function call messages during the current completion call's prompt. The main intuition is to try resuing the generate call logic and it also only appends function call response (comp) without intermediate messages into the final text/messages.

Within interpreter.py

• Updated _execute_gen() logic to include building function call messages if tools are provided, and handle both parallel function calling and non-parallel function calling by either calling backend.generate one time for parallel function call supported models, or multiple times if parallel call is not supported.

[Ying1123]

keep in mind that the set of models that support function calling and parallel function calling are different.

[Yiyun-Liang]

Thanks for pointing out! Updated to have different handling logic.

[Ying1123]

In this case, assert tool_choice is in names of candidate functions.

[Ying1123]

I think it is better to put the logic of real function call into the interpreter, so that it can be reused when we develop the feature for local models.
And remember to handle the logic of appending s.messages_ and s.text_ in the interpreter.

[Yiyun-Liang]

Makes sense, returning just the function call messages here so we can do real function call separately.

[Ying1123]

I think yes, by default. Although accumulating functions could be an option.

[Ying1123]

A design suggestion is that it might be better to just have sgl.gen with func_call as an argument.

[Yiyun-Liang]

I agree, I think it's more straightforward to have it as part of sgl.gen. Would it make sense to have something like sgl.gen("answer_1", max_tokens=256, sgl.func_call(...)) or simply expose parameters directly to sgl.gen like sgl.gen("answer_1", max_tokens=256, tools=[...])?

[Ying1123]

Let's simply expose parameters directly to sgl.gen.

[merrymercy]

https://docs.together.ai/docs/llama-3-function-calling

[felixdivo]

Any updates on this? :)

[Tushar-ml]

It is now supported #2544

[merrymercy]

Close this due to inactivity and #2544.
We can revisit the tool support for the language frontend later.