Wrong serialization of tools / function description field when using chat completion function calling

[the-gigi]

Description

According to the OpenAI API documentation https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools

The format for each tool object is to have two required fields: type and function

Each function object has name (required), description and parameters fields.

However openai-kotlin mixes them up a bit and put the description field at the tool level.

Now, this format works when actually hitting the OpenAI endpoint at https://api.openai.com/v1/

But, it fails when hitting another OpenAI-compatible provider Anyscale at https://api.endpoints.anyscale.com/v1/

You get 400 error:

HttpClient: REQUEST: https://api.endpoints.anyscale.com/v1/chat/completions
METHOD: HttpMethod(value=POST)
COMMON HEADERS
-> Accept: application/json
-> Accept-Charset: UTF-8
-> Authorization: ***
CONTENT HEADERS
-> Content-Length: 589
-> Content-Type: application/json
HttpClient: RESPONSE: 400 
METHOD: HttpMethod(value=POST)
FROM: https://api.endpoints.anyscale.com/v1/chat/completions
COMMON HEADERS
-> content-length: 705
-> content-type: application/json
-> date: Thu, 25 Jan 2024 21:06:00 GMT
-> server: uvicorn
-> via: 1.1 38ad358fbfea219ebf363749e56a6f52.cloudfront.net (CloudFront)
-> x-amz-cf-id: yzeL3R-bGSdNmTR8Wpray0oB-ABQ30c0zM0SHN_ocQ9vf_rLiTO9Nw==
-> x-amz-cf-pop: LAX3-C4
-> x-cache: Error from cloudfront
-> x-request-id: 78b32b97-e074-43d6-91af-edc80db4d50e
HttpClient: RESPONSE https://api.endpoints.anyscale.com/v1/chat/completions failed with exception: io.ktor.serialization.JsonConvertException: Unexpected JSON token at offset 635: Expected beginning of the string, but got { at path: $.error.param
JSON input: ....."OpenAIHTTPException","param":{}},"num_total_tokens":0,"num_.....

It seems that the official OpenAI API server is more forgiving/flexible and can work with the incorrect payload format where Anyscale is more strict (my guess is that they try to serialize the tool into an object that doesn't have a description and the extra description field causes the failure).

Note that using the OpenAI Python library works just fine, so Anyscale does support proper function calling.

Here is the JSON sent by openai-kotlin (description field at tool level)

{'messages': [{'content': ' joke', 'role': 'user'}],
 'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1',
 'tool_choice': 'auto',
 'tools': [{'description': 'this function returns employees and their past '
                           'companies',
            'function': {'name': 'get_company_info',
                         'parameters': {'properties': {'companyName': {'description': 'the company name (e.g Netflix)',
                                                                       'type': 'string'}},
                                        'required': ['companyName'],
                                        'type': 'object'}},
            'type': 'function'}]}

Here is the JSON sent with the OpenAI Python library (different function, but see the format and the description field being inside the function).

{'messages': [{'content': 'You are helpful assistant.', 'role': 'system'},
              {'content': "What's the weather like in San Francisco?",
               'role': 'user'}],
 'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1',
 'tool_choice': 'auto',
 'tools': [{'function': {'description': 'Get the current weather in a given '
                                        'location',
                         'name': 'get_current_weather',
                         'parameters': {'properties': {'location': {'description': 'The city and state e.g. San Francissco, CA',
                                                                    'type': 'string'},
                                                       'unit': {'enum': ['celsius',
                                                                         'fahrenheit'],
                                                                'type': 'string'}},
                                        'required': ['location'],
                                        'type': 'object'}},
            'type': 'function'}]}

Steps to Reproduce

You'll need to register an account and get an API token from Anyscale (you get 10$ worth of free tokens).
See https://docs.anyscale.com/endpoints/model-serving/get-started

Then,
Hit Anyscale endpoint (https://api.endpoints.anyscale.com/v1/) with a chat completion request that has tools.
You can also run the same request against OpenAI https://api.openai.com/v1/ and see that it works

I have a project that demonstrates the problem here:
https://github.com/the-gigi/llm-playground-java

Environment

• openai-kotlin version: 3.6.3
• Kotlin version: 1.9.21
• OS: macOS

Additional Info

Add any other relevant context here.

[the-gigi]

@aallam thanks for fixing this 👏

When do you plan to cut a new release?