Function calling example notebook NoneType error

[a2975667]

The bug
v0.0.64 update for function calling does not seem to be working. It seems like guidance tries to pull content before the function being called.

To Reproduce
In a single python file:

import guidance
import json
from system.setup import setup_chat_llm

guidance.llm = guidance.llms.OpenAI("gpt-3.5-turbo-0613", caching=False, api_key='my_api_key')
print("running guidance version: ", guidance.__version__)

def get_current_weather(location, unit="fahrenheit"):
    """ Get the current weather in a given location.
    
    Parameters
    ----------
    location : string
        The city and state, e.g. San Francisco, CA
    unit : "celsius" or "fahrenheit"
    """
    weather_info = {
        "location": location,
        "temperature": "71",
        "unit": unit,
        "forecast": ["sunny", "windy"],
    }
    return json.dumps(weather_info)



# define a guidance program that uses tools
program = guidance("""
{{~#system~}}
You are a helpful assistant.
{{>tool_def functions=functions}}
{{~/system~}}

{{~#user~}}
Get the current weather in New York City.
{{~/user~}}

{{~#each range(10)~}}
    {{~#assistant~}}
    {{gen 'answer' max_tokens=50 function_call="auto"}}
    {{~/assistant~}}

    {{#if not callable(answer)}}{{break}}{{/if}}
    
    {{~#function name=answer.__name__~}}
    {{answer()}}
    {{~/function~}}
{{~/each~}}""") # type: ignore

executed_program = program(functions=[
    {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA",
                },
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
        }
    }
], get_current_weather=get_current_weather)

print(executed_program["answer"])

The code will throw an error from _gen.py

Traceback (most recent call last):
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/_program_executor.py", line 109, in run
    await self.visit(self.parse_tree, VariableStack([self.program._variables], self))
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/_program_executor.py", line 524, in visit
    command_output = await command_function(*positional_args, **named_args)
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/library/_each.py", line 93, in each
    new_content += await parser.visit(
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/_program_executor.py", line 524, in visit
    command_output = await command_function(*positional_args, **named_args)
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/library/_assistant.py", line 13, in assistant
    return await role(role_name="assistant", hidden=hidden, _parser_context=_parser_context, **kwargs)
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/library/_role.py", line 17, in role
    new_content += await parser.visit(
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/_program_executor.py", line 266, in visit
    visited_children = [await self.visit(child, variable_stack, next_node, next_next_node, prev_node, node, parent_node) for child in node]
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/_program_executor.py", line 266, in <listcomp>
    visited_children = [await self.visit(child, variable_stack, next_node, next_next_node, prev_node, node, parent_node) for child in node]
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/_program_executor.py", line 379, in visit
    command_output = await command_function(*positional_args, **named_args)
  File "/home/tcheng/neo2/.venv/lib/python3.10/site-packages/guidance/library/_gen.py", line 173, in gen
    generated_value += new_text
TypeError: can only concatenate str (not "NoneType") to str

I've tried to see what new_text is from resp

{
  "id": "<some id>",
  "object": "chat.completion",
  "created": 1687315088,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "function_call": {
          "name": "get_current_weather",
          "arguments": "{\n  \"location\": \"New York City\"\n}"
        }
      },
      "finish_reason": "function_call",
      "text": null
    }
  ],
  "usage": {
    "prompt_tokens": 154,
    "completion_tokens": 18,
    "total_tokens": 172
  }
}

My understanding from OpenAI's API, the response need to be sent back to OpenAI's API after the function is being called locally and this does not seem to be the case here?

Thank you again so much for putting this together.

System info (please complete the following information):

• OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): WSL2
• Guidance Version (guidance.__version__): 0.0.64

[xnohat]

I found workaround for this error

set guidance silent=False to make function call working. I still debug why this feature not working in silent mode

[a2975667]

I have not checked, but I think the error can be related to #290

[marcotcr]

Should be working fine in the new release, where function calling is way better. Sorry it took us so long to get to this :)